Created By: Zalmotek

Public Project Link:

https://studio.edgeimpulse.com/studio/139844

Introduction

Glass/window breaking detection systems are used in office buildings for safety purposes. They can be used to detect when a window is broken and trigger an alarm. These systems can also be used to collect data about the event, such as the time, location, and type of break, thus generating data that can be used to further bolster the safety of office buildings in the future.

There are many different types of glass/window breaking detection systems available on the market but they fall in two broad categories:

1. Systems that use vibration and audio sensors to detect the sound of breaking glass.

2. Computer vision based systems used to detect signs of damage in the windows.

The challenge

The biggest challenge with any detection system is to minimize false positives - that is, to avoid triggering an alarm when there is no actual danger. This is especially important in the case of glass/window breaking detection systems, as a false positive can cause significant disruption and even panic.

There are many factors that can cause a false positive with these types of systems, such as:

• Background noise: office buildings are typically full of ambient noise (e.g. people talking, computers humming, etc.) which can make it difficult for sensors to accurately identify the sound of breaking glass.

• Weather: windy conditions can also create background noise that can interfere with sensor accuracy.

• Sound Volume: if the sound of breaking glass is not loud enough, it may not be picked up by sensors.

The Solution

Our approach for those challenges is to create an IoT system based on the Nordic Thingy:53™,development board that will run a machine learning model trained using the Edge Impulse platform that can detect the sound of breaking glass and send a notification via Bluetooth when this event is detected. We have narrowed our hardware selection to the Nordic Thingy:53™ as it integrates multiple sensors (including an accelerometer, gyroscope, microphone, and temperature sensor) onto a single board, which will simplify our data collection process. In addition, the Nordic Thingy:53™ has built-in Bluetooth Low Energy (BLE) connectivity, which will allow us to easily send notifications to nearby smartphones or other devices when our glass/window breaking detection system is triggered. The Nordic Thingy:53 is powered by the nRF5340 SoC, Nordic Semiconductor’s flagship dual-core wireless SoC that combines an Arm® Cortex®-M33 CPU with a state-of-the-art floating point unit (FPU) and Machine Learning (ML) accelerator. This will enable us to run our machine learning model locally on the Thingy:53, without needing to send data to the cloud for processing.

To build our machine learning model, we will be using the Edge Impulse platform. Edge Impulse is a Machine Learning platform that enables you to build custom models that can run on embedded devices, such as the Nordic Thingy:53™. With Edge Impulse, you can collect data from sensors, process this data using various types of Machine Learning algorithms (such as classification or regression), and then deploy your trained model onto your target device.

Edge Impulse has many benefits, the most useful being that you don't need extensive data to train a high-functioning AI model. You can also easily adjust the models based on various needs like processing power or energy consumption.

Hardware Requirements

• Nordic Thingy:53

• Android/iOS device

Software Requirements

• nRF Programmer Android/IoS App

• Edge Impulse account

• Edge Impulse CLI

• Git

Hardware Setup

Due to the fact that the Nordic Thingy:53 comes with a high quality MEMS microphone embedded on it, there is no wiring that must be done. Simply connect the development board to a power supply and move over to the next step.

Software Setup

Creating an Edge Impulse Project

Let's start by creating an Edge Impulse project. Select Developer as your project type, click Create a new project, and give it a memorable name.

Connecting the Device

New Thingy:53 devices will function with the Nordic nRF Edge Impulse iPhone and Android apps as well as the Edge Impulse Studio right out of the box.

Before connecting it to the Edge Impulse project, the firmware of the Thingy:53 must be updated. Download the nRF Programmer mobile application for Android or iOS and launch it. You will be prompted with a number of available samples.

Select the Edge Impulse application, select the version of the sample from the drop-down menu and tap Download.

When that is done, tap Install. A list with the nearby devices will appear and you must select your development board from the list. Once that is done, the upload process will begin.

With the firmware updated, connect the Thingy:53 board to a computer that has the edge-impulse-cli suite installed, turn it on, launch a terminal and run:

edge-impulse-daemon --clean

You will be required to provide your username and password before choosing the project to which you want to attach the device.

Edge Impulse serial daemon v1.14.10
? What is your user name or e-mail address (edgeimpulse.com)? <your user>
? What is your password? [hidden]

Once you select the project and the connection is successful, the board will show up in the Devices tab of your project.

Building the Dataset

For this particular use case, recording data containing glass breaking sounds is challenging. For such situations, Edge Impulse offers its users the possibility of uploading publicly available recordings of various phenomena that can be post-processed in the Data Acquisition tab.

We have gathered over 15 minutes of glass shattering sounds from various license-free SFX sound effects websites and we have uploaded them in our training data pool, using GlassBreaking as their Label. This can be done by navigating to the Upload data tab.

We also need audio for this application that doesn't contain the sound events that we want to identify. We must add sounds that belong to the "background sounds" category to the data pool, such as honks, people talking loudly, doors closing and other various background sounds that the system might be exposed to during normal use. The name of this class should be "BACKGROUND." When populating your dataset, keep in mind that the most crucial component of machine learning is data, and the richer and more varied your data set is, the better your model will perform.

Designing the Impulse

Now that the data is available, it’s time to create the Impulse. The functional Block of the Edge Impulse ecosystem is called “Impulse” and it fundamentally describes a collection of blocks through which data flows, starting from the ingestion phase and up to outputting the features.

The setup is rather straightforward for this use case. We will be using a 1000ms window size, with a window increase of 200ms at an acquisition frequency of 100Hz. For the processing block we will be using an Audio (MFE) and for the Learning block, we will be employing a basic Classification (Keras).

Configuring the Spectrogram Block

When navigating to this menu, you will notice that in the top part of the screen you can explore the time domain representation of the data you have gathered.

Underneath, various parameters of the processing block may be modified. For the moment, we will be moving forward with the default values.

And finally, on the right side of the window you can observe the results of the Digital Signal Process and a spectrogram of the raw signal.

A good rule of thumb when tweaking the DSP block parameters is that similar signals should yield similar results.

Once you are happy with the results, click on Save parameters. After the "Generate features" page loads, click on Generate Features.

In the Generate features tab, you can observe the Feature explorer. It enables visual, intuitive data exploration. Before beginning to train the model, you can rapidly verify whether your data separates neatly. If you're looking to identify the outliers in your dataset, this feature is fantastic because it color-codes comparable data and enables you to track it back to the sample it originated from by just clicking on the data item.

Configure the NN Classifier

The next step in developing our machine learning algorithm is configuring the NN classifier block. There are various parameters that can be changed: the Number of training cycles, the Learning rate, the Validation set size and to enable the Auto-balance dataset function. They allow you to control the number of epochs to train the NN on, how fast it learns and the percent of samples from the training dataset used for validation. Underneath, the architecture of the NN is described. For the moment, leave everything as is and press Start training.

The training will be assigned to a cluster and when the process ends, the training performance tab will be displayed. Here, you can evaluate the Accuracy and the Loss of your model, the right and wrong responses provided by our model after it was fed the previously acquired data set, in a tabulated form.

Moreover, you can see the Data explorer that offers an intuitive representation of the classification and underneath it, the predicted on-device performance of the NN.

Upload the Impulse via USB Cable

You will notice that another menu pops up that allows you to opt in if you want to enable EON Compiler. We will get back to this later, for now click Build and wait for the process to end. Once it’s done, download the .hex file and follow the steps in the video that shows up to upload it on the Thingy:53 board.

With the impulse uploaded, connect the board to your computer, launch a terminal and issue the following command to see the results of the inferencing:

edge-impulse-run-impulse

Upload the Impulse via Android/IoS App

Another way of deploying the model on the edge is using the Nordic nRF Edge Impulse app for iPhone or Android:

1. Download and install the app for your Android/IoS device.

2. Launch it and login with your edgeimpulse.com credentials.

3. Select your project from the list

Navigate to the Devices tab and connect to the Thingy:53

Navigate to the Data tab and press connect. You will see the status on the button changing from Connect to Disconnect.

Navigate to the deployment tab and press deploy.

In the inferencing tab, you will see the results of the Edge Impulse model you have flashed on the device:

Conclusion

In this article, we have described how to create a glass/window breaking detection system using the Nordic Thingy:53™ development board and Edge Impulse Machine Learning platform. This system can be used in office buildings or other commercial settings to help improve safety and security. We believe that this approach has several advantages over existing solutions, including its low cost, ease of use, and accuracy. With further development, this system could be expanded to include other types of sensors (e.g. cameras) to improve detection accuracy or be used in other applications such as door/window opening detection or intruder detection.

Created By: Zalmotek

Public Project Link:

https://studio.edgeimpulse.com/studio/145818

GitHub Repository:

https://github.com/Zalmotek/edge-impulse-appliance-control-voice-nordic-thingy53

Introduction

In today's world, voice commands are becoming a popular user input method for various devices, including smart home appliances. While classical UI methods like physical buttons or a remote control will not be soon displaced, the convenience of using voice commands to control an appliance when multitasking or when having your hands busy with something else, like cooking, cannot be denied.

The Challenge

While very convenient in day to day use, using human speech as user input comes with a number of challenges that must be addressed.

First and foremost, using human speech as user input for smart appliances requires the recognition and understanding of natural language. This means that there must be some sort of keyword detection or voice recognition technology involved. As each user may have their own unique way of phrasing a request, like “ turn on the fan” or “start a 5 minute timer”, the voice recognition algorithms must be fine tuned to obtain the best accuracy.

Another big challenge when implementing such technologies is the security of the gathered data. Privacy concerns regarding speech recognition require cutting-edge encryption techniques to protect voice data and ensure the privacy of any sensitive personal or corporate information transmitted. An easy way to circumvent this is by employing IoT devices that run a machine learning algorithm on the edge and which do not store any data while running the detection algorithm.

Our Solution

We will be demonstrating the design and build process of a system dedicated to integrating basic voice control functionality in any device by using Nordic Thingy:53 dedicated hardware and an audio categorization model developed and optimized using the Edge Impulse platform.

The Nordic Thingy:53™ is an IoT prototyping platform that enables users to create prototypes and proofs of concept without the need for custom hardware. The Thingy:53 is built around the nRF5340 SoC, Nordic Semiconductor’s flagship dual-core wireless SoC. Its dual Arm Cortex-M33 processors provide ample processing power and memory size to run embedded machine learning (ML) models directly on the device with no constraints.

To build the machine learning model responsible for speech recognition we will be using the Edge Impulse platform. Between the plethora of advantages of using this platform, it’s worth mentioning that it does not require a lot of data to train a performant AI model and that it provides a great number of processing power and energy consumption optimization tools, allowing users to build models that can run even on resource restricted devices.

For this use case, we will be using the Nordic Thingy:53 as an advertising peripheral Bluetooth device that listens for the user input and then sends a message via BLE to the ESP32 connected to a relay that can switch on or off an appliance. This system architecture enables users to control multiple appliances distributed around the house, and only one “gateway” that runs the machine learning model on the edge.

Hardware requirements

• Nordic Thingy:53

• Esp32 DevKit

• Relay

• Android/iOS device

• J-link mini Edu

• 220V to 5V power regulator

• Plug-in plastic enclosure

Software requirements

• Edge Impulse account

• Edge Impulse CLI

• Arduino IDE

• Arduino CLI

• Git

• A working Zephyr environment

Hardware Setup

To control appliances you will either have to work with AC mains or integrate with some functionality that the appliance might have, such as IR control or switches.

We chose to connect the Adafruit Non-Latching Mini Relay FeatherWing to the ESP32 development board as presented in the following schematic. The Signal pin of the relay is connected to the GPIO32 pin of the ESP32 and the ground is common between the ESP32 and the relay.

You can use this circuit to control AC powered household devices, such as kettles, lights, or stove smoke extractors.

We have chosen an enclosure that will safely protect users from the AC Mains and the rest of the electronics. We have soldered the circuit on a test board by using the following schematic that we tested first on a breadboard. We have kept the testboard neatly separated between low DC voltage that was routed in the top part and AC High voltage that was routed in the lower part of the testboard.

Warning: Working with AC mains is dangerous if you have never done it before, please ask for an electronics senior or document yourself thoroughly before undergoing this schematic.

The enclosure provides an AC In and an AC Out socket plug that allows us to integrate our electronics between thus keeping the test boards continuously supplied and enabling or disabling the output thus turning on or off the supply.

Use the appropriate wire gauge when working with AC so the current draw of the appliance is met. We have used standard 16A wire gauge with the proper colors (blue for neutral, brown for line and yellow/green for grounding according to EU standards).

The gateway can be placed anywhere in the house and does not need to be connected to a power outlet as it is battery powered, making it much more convenient for users.

Software Setup

Setup the Build Environment

Building and flashing custom applications on the Nordic Thingy:53 board require a working Zephyr environment. To create it, follow the steps in the Getting Started guide from the official Zephyr documentation. Afterwards, follow the steps presented in the Developing with Thingy:53 guide from the official Nordic Semiconductor documentation. While this might not be mentioned in either of the documents, you must also install the J-Link Software and Documentation Pack and the nRF Command Line Tools(ver 10.15.4) to be able to flash the board.

After following the steps in the guides presented above, you should have a working Zephyr environment. Remember to always work in the virtual environment created during the Getting Started guide when developing applications for this platform.

Creating an Edge Impulse Project

Let's start by creating an Edge Impulse project. Select Developer as your project type, click Create a new project, and give it a significant name.

Connecting the Device

Thingy:53 devices will work with the Nordic nRF Edge Impulse iPhone and Android apps or with the Edge Impulse Studio right away.

First of all, the firmware of the Thingy:53 device must be updated. Download the nRF Programmer mobile application and launch it. You will be prompted with a number of available samples.

Select the Edge Impulse application, select the version of the sample from the drop-down menu and tap Download.

Once that is done, tap Install and a list with nearby devices will appear. You have to select your development board from the list and the upload process will begin.

With the firmware updated, connect the Thingy:53 board to a computer that has the edge-impulse-cli suite installed, turn it on, launch a terminal and run:

edge-impulse-daemon --clean

You will be required to provide your username and password before choosing the project to which you want to attach the device.

Edge Impulse serial daemon v1.14.10
? What is your user name or e-mail address (edgeimpulse.com)? <your user>
? What is your password? [hidden]

Once you select the project and the connection is successful, the board will show up in the Devices tab of your project.

Building the Dataset

Considering the context, the best way to gather relevant data for your Thingy:53 is to record yourself. To achieve the best performance of the speech recognition model, you can add samples of your own voice to increase the specificity of the detection algorithm.

Go to Data Acquisition in your Edge Impulse project and you can start gathering the data set.

For this particular use case, we will be using the "Light", "Kettle" and "Extractor" keywords to turn on different items in the kitchen. Now start recording 5-10 seconds segments of you saying "Light", "Kettle" and "Extractor". You will notice that they appear in the Collected data tab. Click on the menu symbolized by three points and press Split Sample. Edge Impulse automatically splits the sample in 1 second windows but you can adjust those manually. When you are happy with the windows, press Split.

We will also require audio that doesn't contain the keywords we wish to detect. We must gather a data set with sounds like ambient noise, people speaking in the distance, or different sound from the kitchen all of which fall within the "background" category. This class is identified as "Background". Keep in mind that data is the most important part of machine learning and your model will perform better the more diverse and abundant your data set is.

From now on, because "Light", "Kettle" and "Extractor" are the classes we wish to detect, we will be referring to them as "positive classes" and to "Background" as a negative class. What we will be doing is use the keywords dataset from the Unknown and Noise samples for background noises.

Download the dataset and navigate to the Upload data menu. We will be uploading all the samples to the Training category and we will label the "Noise" and "Unknown" samples with the label "Background".

Designing the Impulse

Let's begin developing our Impulse now that the data are available. An Impulse describes a group of blocks through which data flows and can be viewed as the functional Block of the Edge Impulse ecosystem.

The input level, the signal processing level, the learning level, and the output level are the four levels that make up an impulse.

For the input block, we will leave the setting as default. As for the processing and learning block, we have opted for an Audio (MFCC) block and a basic Classification (Keras).

Do the set up just like in the image above and click on Save Impulse and move on to configure the blocks one by one.

Configuring the Audio Features Block

The Audio MFCC (Mel Frequency Cepstral Coefficients) block extracts coefficients from an audio signal using Mel-scale, a non-linear scale. This block is used for human voice recognition, but can also perform well for some non-voice audio use cases. You can read more about how this block works here.

You can use the default values for configuring the MFCC block and click on Save parameters. You’ll be prompted to the feature generation page. Click on Generate features and you will be able to visualise them in the Feature explorer.

Configure the Classifier (NN)

The next step in developing our machine learning algorithm is configuring the NN classifier block. Before training the model, you have the opportunity to modify the Number of training cycles, the Learning rate, the Validation set size and to enable the Auto-balance dataset function. The learning rate defines how quickly the NN learns, the number of training cycles determines the number of epochs to train the NN on, and the size of the validation set sets the percentage of samples from the training data pool utilized for validation. You can leave everything as it is for now and press Start training.

The training will be assigned to a cluster and when the process ends, the training performance tab will be displayed. There will be displayed the Accuracy and the Loss of the model, as well as the right and wrong responses provided by the model. You can also see an intuitive representation of the classification and underneath it, the predicted on-device performance of the NN.

Test the Impulse Using iOS / Android

One way of deploying the model on the edge is using the Nordic nRF Edge Impulse app for iPhone or Android:

1. Download and install the app for your Android/IoS device.

2. Launch it and log in with your edgeimpulse.com credentials.

3. Select your Smart Appliance Control Using Voice Commands project from the list

1. Navigate to the Devices tab and connect to the Thingy:53:

1. Navigate to the Data tab and press Connect. You will see the status on the button changing from Connect to Disconnect.

1. Navigate to the Deployment tab and press Deploy.

1. In the Inferencing tab, you will see the results of the Edge Impulse model you have flashed on the device:

Creating a Custom Application

To showcase the process of creating a custom application, we have decided to create a basic Bluetooth application. In this application, the Thingy:53 functions as a peripheral bluetooth device that advertises itself. The ESP32 functions as a bluetooth client that scans for available devices to connect to. When it detects the Thingy:53, it pairs with it and awaits a command.

After the devices are paired, when the central button of the Thingy:53 is pressed, it sends a message to the Esp32 which triggers a Relay.

Here you can find the source code for the Thingy:53 and the Esp32.

To build and flash the application on the Nordic hardware, copy the folder named Thingy53_Peripheral from the repository, to ncs/nrf folder and then run:

&west build -p always -b thingy53_nrf5340_cpuapp
nrf/Thingy53_Peripheral/

Make sure you have the board powered on and connected via the J-Link mini Edu to your computer and run

&west flash

To flash the esp32, follow the steps provided here to set-up the build environment and then, simply copy the code from the ESP32_Client.ino file in a new sketch and press upload.

Future Development

For this project we have decided to deploy the machine learning algorithm on the Bluetooth peripheral device. This enables the possibility of using multiple Central devices that are dedicated to switching the appliance on and off, while the processing load of running the machine learning algorithm is done by the Thingy:53.

Another great addition to this project would be the implementation of the Matter protocol.

Matter is a royalty free standard and was created to encourage interoperability between different devices and platforms.

If the appliances already have an IoT layer, the Thingy:53 is fully compatible with the Matter and instead of using relays, it could be directly interfaced with the smart appliances.

Conclusion

In this article, we have presented a very basic implementation of Edge Impulse on the Thingy:53 to control an appliance using voice commands. The use of Edge Impulse and the integration with Nordic Semiconductor's IoT platform opens up endless possibilities for creating intelligent and user-friendly appliances.

The ability to quickly gather data, create, train and deploy machine learning algorithms greatly simplifies the process for developers, making it easier for them to incorporate these technologies into their projects.

Ultimately, this system provides a convenient and cost-effective way to control multiple appliances in the home.

We hope that this article will inspire you to try out Edge Impulse and Nordic Semi's Thingy:53 in your own smart appliance projects.

Created By: Zalmotek

Public Project Link:

https://studio.edgeimpulse.com/studio/126774

Introduction

Illegal logging is a major environmental issue worldwide. It not only destroys forests but also decreases the amount of available timber for legal purposes. In addition, illegal logging often takes place in protected areas, which can damage ecosystems and jeopardize the safety of people and wildlife.

One way to combat this problem is through the use of machine learning algorithms that can detect chainsaw noise and be deployed on battery-powered devices, such as sensors in the forest. This allows for real-time monitoring of illegal logging activity, which can then be quickly addressed.

The Challenge

Detecting illegal logging is therefore essential for both environmental and economic reasons. However, it is difficult to detect illegal logging activity due to the vastness of forested areas, as it often takes place in remote and hard-to-reach areas. Traditional methods such as ground patrols are often ineffective, and satellite imagery can be expensive and time-consuming to analyze.

When combined with satellite data, ML can be used to quickly and cost-effectively identify areas where illegal logging is taking place. By deploying sensors in the forest, they are able to detect illegal logging activity in real time. In addition, by providing data on where illegal logging is taking place, they are helping to better target enforcement efforts. This information can then be used to deploy ground patrols or take other actions to stop illegal activity. In this way, ML can play a vital role in protecting forests and ensuring that timber is harvested legally.

Our Solution

Our approach to this problem is to create an IoT system based on the TinyML Syntiant development board that will run a machine learning model trained using the Edge Impulse platform that can detect the sound of chainsaws and send a notification via SMS when this event is detected.

We have picked the Syntiant TinyML board as our specialized silicon that can easily run the machine learning algorithm thanks to its Neural Decision Processors (NDPs). Moreover, having an onboard PH0641LM4H microphone makes it great for quickly prototyping audio-based projects. This board is ideal for building and deploying embedded Machine Learning models, as it is fully integrated with Edge Impulse and it has very low power consumption, advantageous for this use case as we do not want to charge the battery often.

Because this device must be deployed in remote and difficult-to-reach locations, a fitting power source must be used and so, we have opted to power it from a 2500 mAh rechargeable Li-Ion battery.

We will use a publicly available chainsaw noise dataset and the Edge Impulse platform to train and deploy a model that can distinguish between this sound and other similar sounds.

Hardware requirements

• Syntiant TinyML board

• Micro USB cable

• SIM800L GSM module

• ESP32

• ESP32 programmer

• 3.3V power supply (LD1117S33)

• Power bank: rechargeable Li-Ion battery (INR18650-29E6 SAMSUNG SDI) + charger

• Consumables (wires, prototyping board, LED)

• Enclosure (carved stone)

Software requirements

• Edge Impulse account

• Edge Impulse CLI

• Arduino IDE

• Arduino CLI

• Git

Hardware Setup

As stated before, our choice of Edge computing hardware for this use case is the Syntiant TinyML board, designed for working with Syntiant’s Neural Decision Processors (NDPs). This development board is a powerful tool for developing and deploying Machine Learning algorithms, as it is equipped with an ARM Cortex-M4 processor, which allows for real-time inference of the trained Machine Learning model. Another advantage of this Syntiant board is that it is equipped with 5 GPIOs that enable it to interact with other external circuitry and trigger an external output.

Over the ML detection layer of our system, we will be adding a communication layer. In this case, we are using an external circuit based on the ESP32 MCU that can read a trigger from Syntiant TinyML, which is further interfaced with a SIM800L module, responsible for sending an SMS message to the user when chainsaw noise is detected.

Be advised that the SIM800L is using 2G networks, in the EU there are still many providers for it and the coverage is way better for it in comparison with the speedier networks. Since we are deploying it in a forest it's actually a good option for us. In the US the 2G networks are being phased out currently so be sure to check what GPRS module works best in your case.

In order to camouflage the device, we can fit all of the electronics into a fake stone, making it less likely to be detected. This will make it difficult for illegal loggers to find and destroy the device, as they will not be able to see it.

Since we want to fit all the modules inside the rock, we’re using the smaller SMD ESP32-WROOM-32UE SoC instead of the ESP32 development board, so we also have to use a breakout board to program the board.

To power the system we’re using a power bank based on the rechargeable INR18650-29E6 Li-Ion battery, from which we can obtain both 5V for the Syntiant TinyML board and 3.8V for the SIM800L GSM module. We also have to use a 5V to 3.3V voltage regulator to power the ESP32. You can find all the details in the full wiring schematic presented below.

NOTE: The Syntiant TinyML board has no power voltage input (the 5V pin does not power up the board and by looking at the schematic we can see it's actually before a voltage regulator that draws from the USB power supply). We had to resort to soldering a small wire to a 5V test pad on the bottom of the board that luckily powers the board.

Because of the space constraints, we opted for a quick point-to-point wiring solution by directly soldering the components for our proof of concept. Of course, after further testing, a printed circuit board could be designed to accommodate all modules and provide a direct plug for the battery making it easier to produce in larger numbers in an automated fashion.

Software Setup

To connect the Syntiant TinyML to Edge Impulse, download the Audio firmware archive. Put the board in boot mode by double-clicking on the reset button when connecting the board to your computer, while the orange LED is blinking. In boot mode, you should see the red LED fading on and off. Then flash the firmware by running the script for your OS from the archive:

# for macOS and Linux  
./arduino-build.sh --flash
# for Windows
arduino-win-build.bat --flash

Creating an Edge Impulse Project

Let's start by creating an Edge Impulse project.

Register an account with Edge Impulse. Select Developer as your project type, click Create new project, and then give it a memorable name.

Afterward, select Accelerometer data as the type of data you will be dealing with.

Data Collection

Create a new Edge Impulse project, give it a meaningful name, and select Developer as your desired project type. Afterward, select Audio as the type of data you wish to use. To collect audio data from the microphone, connect the board to your computer. Once plugged in, the Syntiant TinyML Board shows up as a USB microphone and Edge Impulse can use this interface to record audio directly. Make sure the board is selected instead of your default microphone in your audio input settings.

Then, go to Devices -> Connect a new device in your Edge Impulse project, choose Use Your Computer, and allow access to your microphone.

Building the Dataset

Sometimes, gathering relevant data for your TinyML model might be a complicated endeavor, just like in this case. This is a situation in which the Upload data function under the Data acquisition tab saves the day. What it allows you to do is use publicly available recordings of the phenomena you wish to identify and use them as training data for your model.

In this case, what you must do is find publicly available recordings of chainsaws (in WAV format) and download them. Navigate to the Upload data menu, click on Choose Files and select your files, select Training as the upload category and manually enter the "Chainsaw" Label and click Begin upload.

It’s important to upload data that corresponds to background or unknown sounds because the NDP101 chip expects one and only negative class and it should be the last in the list, that we will be naming Z_Background.

For this, we will be using the Noise and Unknown dataset that can be found in the Keyword Dataset, which we will label Z_Background.

The next thing on the list is the data formatting. The data uploaded from the Keyword Dataset is split into 1-second long recordings, while the Chainsaw sounds are usually longer, over 30 seconds long. To make things uniform, we will select one Chainsaw data entry, click on the menu marked by the 3 vertical dots, and press on Split Sample. We will leave the segment length to 1000ms and add additional segments if needed. Click on Split and then do this for every chainsaw sample we have uploaded.

Once this is done, it’s time to do the Train/Test Split. Just click on the triangle with an exclamation mark on it, in the Train/Test Split section and Edge Impulse will do the job for you. An ideal ratio would be 80%-20%.

Designing the Impulse

Once the Data acquisition is over, it’s time to build the Impulse.

You will notice the odd window size. Make sure to leave it at the default 968ms because this is a hardware constraint of the NDP101 chip.

For the processing block, we will be using Audio (Syntiant). This will compute log Mel-filterbank energy features from the audio signal that is fed in it and as for the learning block, we will be employing a Classification(Keras) block.

Once you are happy with the set-up, click on Save Impulse.

Configuring the Digital Processing Block

What this submenu does is let you explore the raw data, and see the signal processing block’s results.

Navigate to the "Syntiant" submenu under Impulse Design, leave the parameters on their default value and click on Save parameters, and then on Generate Features.

One of the most effective tools provided by Edge Impulse is the Feature Explorer. It enables visual, intuitive data exploration that enables you to rapidly verify whether your data separates neatly, even before starting to train your model. If you're looking to identify the outliers in your dataset, this feature is fantastic because it color-codes comparable data and enables you to track it back to the sample it originated from by just clicking on the data item.

Configure the Neural Network

In this stage, we may provide a number of parameters that affect how the neural network trains under the NN Classifier tab in the Impulse Design menu. The Training set can now be kept at its default setting. When you click the Start Training button, take note of how a processing cluster is chosen to host the training process.

What is worth noting here is the particular structure of the neural network. To be more precise, we need to use a 3-layer NN architecture, every layer consisting of 256 neurons to be able to run our model on the NDP101 chip.

Once the program is finished, you will see the training output displayed. A degree of accuracy of more than 95% is what we aim for. The accurate and incorrect responses that our model provided after being fed the previously gathered data set are shown in tabular form in the Confusion matrix just below it. In this case, the dataset was clean enough that the model exhibits a 100% Accuracy rating.

Model Testing

Navigating to the Model Testing page is a wonderful method to start testing our model before deployment. The samples kept in the Testing data pool will be displayed to you. To run all of this data through your impulse, select Classify all.

Before making the effort to deploy the model back on the edge, the user has the opportunity to test and improve the model using the model testing tab. When building an edge computing application, the ability to go back and add training data, modify the DSP and Learning block, and fine-tune the model reduces development time significantly. We strongly recommend taking the time to go back, tweak and optimize your model before deploying it on the edge.

To check how the model behaves when deployed on the edge, navigate to the Deploy tab, click on Syntiant TinyML under the Build firmware section, select the posterior parameters as the audio events we wish to detect, and press build.

Edge Impulse will then compile a pre-built binary that can be deployed in the same manner as the data-forwarder used in the data acquisition phase.

With the model uploaded on the board, you can use any serial monitor, like picocom or the serial monitor embedded in Arduino IDE to debug and evaluate the performance of the Impulse.

BIN File Loaded correctly from Serial Flash Setup for audio done
Loaded configuration
Inferencing settings:
	Interval: 0.0625 ms.
	Frame size: 15488
	Sample length: 968 ms.
	No. of classes: 2
Starting inferencing, press 'b' to break
Predictions:
    Chainsaw: 	0
    Z_Background: 	1

Deploying the Model on the Edge

Once you are happy with your model, it's time to deploy it on the device.

For our use case, we will be using our custom machine learning model alongside the firmware provided by Edge Impulse to create a custom application that will trigger an output when Chainsaw sounds are detected.

First and foremost, navigate to the Deployment tab, select Syntiant NDP101 library, and then click on "Find posterior parameters". This is a particularity of the NDP101 architecture that allows the user to set the thresholds for words or other audio events at which a model activates.

Select "Chainsaw" as the audio event that you want to detect and press Find parameters. Once that is done, you are ready to press Build.

The task will be assigned to a computing cluster and when the job finishes, you will be able to download it locally.

Afterward, clone the Syntiant firmware repository and copy and paste the model files into the "src" folder.

Open the "firmware-syntiant-tinyml.ino" in Arduino IDE, without creating a folder for it, and modify the on_classification_changed function like so:

#include "src/syntiant.h"
#include <NDP.h>
#include <NDP_utils.h>
#include <Arduino.h>

void on_classification_changed(const char *event, float confidence, float anomaly_score) {
    if (strcmp(event, "Chainsaw") == 0) {
        OUT_1_LOW();
        delay(10000);
    }else{
        OUT_1_HIGH();
    }
}
void setup(void)
{   syntiant_setup();
    OUT_1_HIGH();
}
void loop(void)
{
    syntiant_loop();
}

Save and close.

Next up, launch a terminal, navigate to the firmware-syntiant-tinyml folder and run:

./arduino-build.sh --build

Once the firmware successfully builds, connect the Syntiant board to your computer, place it in Boot mode and run:

./arduino-build.sh --flash

If everything goes smoothly, the firmware will be flashed on your Syntiant TinyML board and when your board detects Chainsaw sounds, it will light up the LED in the color red.

Sending SMS Notifications

Now that you have a Syntiant TinyML board that is able to detect chainsaw noises, you can program the ESP32 board to send an SMS notification through the SIM800L GSM module whenever the sound is detected. The GSM module is connected to the UART2 of the ESP32 microcontroller. Open Arduino IDE and paste the code below, then adjust the phone number and upload it to your ESP32 board:

// code adapted from: https://circuitdigest.com/microcontroller-projects/interfacing-sim800l-module-with-esp32

void setup() {
  Serial.begin(9600);
  Serial2.begin(9600);
  delay(3000);
  pinMode(16, INPUT);
  pinMode(18, OUTPUT);
  digitalWrite(18, LOW);
  test_sim800_module();
}

void loop() {
  updateSerial();
  if (digitalRead(16) == HIGH) {
    digitalWrite(18, HIGH);
    send_SMS();
    delay(1000);
  }
  digitalWrite(18, LOW);
}

void test_sim800_module() {
  Serial2.println("AT");
  updateSerial();
  Serial.println();
  Serial2.println("AT+CSQ");
  updateSerial();
  Serial2.println("AT+CCID");
  updateSerial();
  Serial2.println("AT+CREG?");
  updateSerial();
  Serial2.println("ATI");
  updateSerial();
  Serial2.println("AT+CBC");
  updateSerial();
}

void updateSerial() {
  delay(500);
  while (Serial.available()) {
    Serial2.write(Serial.read()); // Forward what Serial received to Software Serial Port
  }
  while (Serial2.available()) {
    Serial.write(Serial2.read()); // Forward what Software Serial received to Serial Port
  }
}

void send_SMS() {
  Serial2.println("AT+CMGF=1"); // Configuring TEXT mode
  updateSerial();
  Serial2.println("AT+CMGS=\"+xxxxxxxxxxx\""); // change ZZ with country code and xxxxxxxxxxx with phone number to sms
  updateSerial();
  Serial2.print("Illegal logger detected!"); // SMS
  updateSerial();
  Serial.println();
  Serial.println("Message Sent");
  Serial2.write(26);
}

Conclusion

In conclusion, illegal logging is a major environmental issue that can be effectively combated through the use of machine learning algorithms. The system described in this article provides a way to quickly and easily monitor large areas of forest for illegal logging activity, and take action to stop it. The Syntiant TinyML board is an efficient and robust platform for running machine learning models and can be used to quickly and easily detect illegal logging activity. With this system in place, we can help to preserve our forests and ensure that they are managed in a sustainable way.

If you need assistance in deploying your own solutions or more information about the tutorial above please reach out to us!

Created By: Wamiq Raza

Public Project Link: https://studio.edgeimpulse.com/public/109559/latest

Story

Snoring, a type of sleep disordered breathing, disrupts sleep quality and quantity for both the snorer and, frequently, the person who sleeps with the snorer. Snoring-related sleep deprivation can cause serious physical, emotional, and financial difficulties. Snoring not only disrupts the snorer's sleep, but it can also cause anger between spouses! About 40% of adult males and 24% of adult women snore on a regular basis. Snoring begins when the muscles around the throat relax during sleeping. This narrows the airway, causing vibrations that result in snoring. Snoring is more common when a person sleeps on their back. Sleeping on your side is a natural remedy for snoring. Sleeping on your side rather than your back is a simple and natural treatment for snoring. In this project, a deep learning model for snoring detection is designed to be implemented on a smart phone using the Edge Impulse API, and the model may be put on other embedded systems to detect snoring automatically. A smart phone is linked to the listener module by home Wi-Fi or mobile data to log snoring incidents with timestamps, and the data may be shared to a physician for treatment and monitoring of disorders such as sleep apnea.

Getting started

This tutorial has the following requirements:

• Basic understanding of software development

• Edge Impulse account

• Android or iOS mobile phone

We will use Edge Impulse, an online development platform for machine learning on edge devices. Create a free account by signing up here. Log into your account and give your new project a name by clicking on the title. To run the current project as-is, you can directly clone it to make your own copy, and start executing.

Data Acquisition

This project used a dataset of 1000 sound samples which is divided into two categories: snoring noises and non-snoring sounds. There are 500 examples in each class, the snoring noises were gathered from several web sources and non-snoring sound were gathered from similar web sources. The files were then separated into equal sized one second duration files after silences were removed from them. As a result, each sample lasts one second. Among the snoring samples, 363 samples consist of snoring sounds of children, adult men and adult women without any background sound [1]. The remaining samples having a background of non-snoring sounds. The 500 non-snoring samples consist of background sounds that are ten categories such as baby crying, the clock ticking, the door opened and closed, total silence and the minor sound of the vibration motor of the gadget, toilet flashing, siren of emergency vehicles, rain and thunderstorms, streetcar sounds, people talking, and background television news. Figure 1 and Figure 2 illustrate the frequency of snoring and non-snoring sound respectively. The dataset can be downloaded from [2].

Once the dataset is ready you can upload it into Edge Impulse. Figure 3 represents the Edge Impulse platform, and how to upload the data. If you don’t want to download or upload data, and just run the project, refer to the section Getting started for cloning the current project.

Creating Your Impulse

Next, we will select signal processing and machine learning blocks, on the Create impulse page. The impulse will start out blank, with Raw data and Output feature blocks. Leave the default settings of a 1000 ms Window size and 1000 ms Window increase as shown in Figure 4. This means our audio data will be processed 1 second at a time, starting each 1 second. Using a small window saves memory on the embedded device.

Click on ‘Add a processing block’ and select the Audio (MFE) block. Next click on ‘Add a learning block’ and select the Neural Network (Keras) block. Click on ‘Save Impulse’ illustrated in Figure 5. The audio block will extract a spectrogram for each window of audio, and the neural network block will be trained to classify the spectrogram as either a 'snoring' or 'no snoring' based on our training dataset. Your resulting impulse will look like this:

Next we will generate features from the training dataset on the MFE page, as shown in Figure 6. This page shows what the extracted spectrogram looks like for each 1 second window from any of the dataset samples. We can leave the parameters on their defaults.

Next click on the ‘Generate features’ button, which then processes the entire training dataset with this processing block. This creates the complete set of features that will be used to train our Neural Network in the next step. Press the ‘Generate features’ button to start the processing, this will take a couple minutes to complete, and the result of the generated features can be seen in Figure 7.

We can now proceed to setup and train our neural network on the NN Classifier page. The default neural network works well for continuous sound. Snoring detection is more complicated, so we will configure a richer network using 2D convolution across the spectrogram of each window. 2D convolution processes the audio spectrogram in a similar way to image classification. Refer to the "NN classifier" section in my project for the architecture structure.

Model Training and Results

To train the model, the number of epochs was set to 100, the leaning rate assigned after several trials is 0.005, and the overall dataset was split into 80% training and 20% validation set. The number of epochs is the number of times the entire dataset is passed through the neural network during training. There is no ideal number for this, and it depends on the data. In Figure 8 we can see the feature explorer for correct and incorrect classification of both classes.

The model confusion matrix and on mobile device performance can be seen in Figure 9. The overall accuracy of the quantized int8 model is 94.3%, with 93.5% 'no snoring' and '95.1%' snoring correct classification, and the rest are misclassified.

After the training model we run the model for testing purposes. It gives us an accuracy of 97.42% on test data, as presented in Figure 10, along with feature exploration.

Live Classification

The Live classification page allows us to test the algorithm both with the existing testing data that came with the dataset, or by streaming audio data from your mobile phone or on any microcontroller with audio data processing compatibility. We can start with a simple test by choosing any of the test samples and pressing ‘Load sample’. This will classify the test sample and show the results: in Figure 11 and 12 respectively for both classes with their probability score. We can also test the algorithm with live data. Start with your mobile phone by refreshing the browser page on your phone. Then select your device in the ‘Classify new data’ section and press ‘Start sampling’ as shown in Figure 15.

Deployment

In order to deploy the model on a smartphone, go to the browser window on your phone and refresh, then press the ‘Switch to classification mode’ button as shown in Figure 13. This will automatically build the project into a Web Assembly package and execute it on your phone continuously (no cloud required after that, you can even go to airplane mode!). Once you scan the QR code on your mobile it will ask you to access the microphone, as in this project we are using audio data, as shown in Figure 14. After the access is granted, we can see the classification results.

Furthermore, if you would like to extend the project, you can run it on different microcontrollers. Of course, for this project, our aim was to provide the insight of TinyML development on a smartphone.

Conclusion

This project provides insights of TinyML deployment on a smartphone. In this project, a deep learning model for snoring detection is trained, validated, and tested. A prototype system comprising of a listener module for snoring detection was demonstrated. For future work, extend the default dataset with your own data and background sounds, remember to retrain periodically, and test. You can set up unit tests under the Testing page to ensure that the model is still working as it is extended.

References

1. T. H. Khan, "A deep learning model for snoring detection and vibration notification using a smart wearable gadget," Electronics, vol. 8, no. 9, article. 987, ISSN 2079-9292, 2019

2. Khan, T. A Deep Learning Model for Snoring Detection and Vibration Notification Using a Smart Wearable Gadget. Electronics 2019, 8, 987. https://doi.org/10.3390/electronics8090987

Created By: Manivannan Sivan

Public Project Link:

Project Demo

Impact of Leakages in Pipes

Water is the world's most precious resource, yet it is also the one which is almost universally mismanaged. As a result, water shortages are becoming ever more common. In the case of water supply and distribution networks, these manifest themselves in the intermittent operation of the system. Not only is this detrimental to the structural condition of the pipes, but can also adversely affect the quality of the water delivered to the customer's taps. Further, leakage often exceeds 50% of the production. Not only does this have a significant economic impact, but an environmental one too. But to recover leakage has a cost to undertake a hydraulic study of the network, create a permanent monitoring system, and eliminate the leaks. So how low should leakage go and how can a lower leakage level be maintained over time? This is the objective of the very innovative EU funded PALM project recently completed in central Italy.

Increase in Carbon Level Due to Water Leakage

There is an increased carbon footprint of having pumps constantly running to make up for the water lost due to leakage. It is the increased pump use, and pump maintenance/replacement costs that increase CO2 in the air from the fossil fuels being burned to support it. According to a study done by Von Sacken in 2001, water utilities are the largest user of electricity accounting for 3% of the total electricity consumption in the US. In addition, it is estimated that 2-3 billion kW/h of electricity is expended pumping water due to leakage.

Costs, health, the environment, and infrastructure are just a few things that can come into play when water system leakage goes uncorrected.

More than 2 billion people globally live in countries with high water stress, per the 2018 statistics provided by the United Nations (UN). In order to tackle this problem, it is necessary to conserve and utilize water safely. Installation of proper water pipeline leak detection systems assist in specifying the leakages in installed water pipes, which ultimately avoids wasting water through cracks and holes. Therefore, the increasing scarcity of water is propelling the demand for water leak solutions, which in turn drives the market.

Global Market for Pipe Leakage Detection Systems

The global water pipeline leak detection systems market size is expected to reach $2,349.6 million in 2027, from $1,748.6 million in 2019, growing at a CAGR of 6.8% from 2020 to 2027. Water pipeline leak detection systems are utilized to determine the location of the leak in water transmission pipelines. Around 30% to 50% of water is lost through aging pipelines, which also contributes toward loss of revenue. Water pipeline leak detection systems are available for both underground and overground water pipelines to precisely locate and check the severity of pipeline leaks.

On the contrary, in recent years, pipeline leak detection systems have undergone various technological advancements by adoption of computerized systems and digital survey systems. The traditional acoustic detection sensors are upgraded with more efficient sound detection functions which has increased their efficiency. Introduction and implementation of such advanced technologies are likely to create lucrative opportunities for the growth of the water pipeline leak detection systems market during the forecast period.

In recent years, the increase in acoustic-based pipe leakage detection has started increasing due to investment in R&D.

A TinyML-based Solution for Pipe Leakage Detection:

In this fast growing sector, TinyML-based systems will play a major role due to low power consumption and developing EdgeML models with more accuracy in predicting leakage detection.

My prototype is based on acoustic data collected on an Arduino Portenta H7 and a model is trained using Edge Impulse. In my prototype, The Arduino Portenta Vision Shield is used because it contains two microphones (MP34DT05) which runs on 16 MHz. The Vision Shield is placed on top of the pipe for data acquisition as the microphone faces the pipe. This will help to collect the noise of the water flowing.

In the data acquisition stage, the pipe is simulated with "Idle" mode, where the tap is fully closed so no water flows, and then slightly opened to simulate "leakage mode". Finally, is it fully opened to simulate "water flow" mode.

Pre-Processing

In a pre processing stage, the Window size is set as 2000ms and Window increase is set as 500ms.

For Neural Network configuration, I have used couple of 1D-Conv layers followed by DNN layers.

The number of Training cycles is set to 100 and Learning rate is set to 0.005. The accuracy obtained was 99.1 % with loss of 0.02 only. As the model is performing well at classifying the data, we can move on.

Model Testing

In Model testing, the trained model is tested with data and it is able to predict all 3 conditions we trained on with 100% accuracy.

Deployment

Then in Deployment section, select Arduino Portenta H7 and download the firmware files to your computer.

Press the Reset button twice on the Portenta to change it to Flash mode. Then run the .bat file if you are on Windows, or the Mac or Linux commands if you are on those platforms.

Summary

The prototype demonstrated an acoustic method to predict leakage in a pipe. The model was able to determine whether the pipe is in Idle (no water flowing), flowing normally, or if there is a small flow, representing leakage in this case. The use case is simple enough to apply to any industry to monitor the leakages in pipe, though this is of course only a prototype project.

By integrating well-designed enclosures with higher quality microphones, the Arduino Portenta H7 will be ideal for industrial use-cases for pipe leakage detection.

Created By: Eivind Holt

Public Project Link:

GitHub Repository:

Project Demo

Intro

This wearable device detects and reports user's coughs. This can be useful in treatment of patients suffering from chronic obstructive pulmonary disease, COPD, a group of diseases that cause airflow blockage and breathing-related problems. The increase in number and intensity of coughs can indicate ineffective treatment. Real-time monitoring enables caregivers to intervene at an early stage.

Advantages over existing solutions

Privacy

Existing methods of analyzing audio recordings greatly invades privacy of the patient, caregivers and peers. This proof-of-concept does not store any audio for more than a fraction of a second. This audio buffer never leaves the device, it is constantly being overwritten as soon as the application has determined if the small fragment of audio contains a cough or not. In fact, the hardware used is not capable of streaming audio using the low-energy network in question.

Further, the application is hard-coded to detect coughs or noise. To be able to detect new keywords, for instance "bomb", or "shopping", the device would have to be physically reprogrammed. Firmware Over-the-Air is not currently supported in this project. Each keyword consumes already constrained memory, limiting the practical amount of different keywords to a handful.

Compared to commercial voice assistants, such as Google Nest, Amazon Alexa or Apple Siri on dedicated devices or on smartphone, this device works a bit differently. The aforementioned products are split into two modes: activation and interpretation. Activation runs continuously locally on the device and is limited to recognizing "Hey google" etc. This puts the device in the next mode, interpretation. In this mode an audio recording is made and transmitted to servers to be processed. This opens up for greatly improved speech recognition. It also opens up to secondary use, better know as targeted advertisement. The device in this project only works in the activation mode.

Hardware, software, tools and services used

• Arduino Nano 33 BLE Sense

• LiPo battery

• JSH battery connectors

• Edge Impulse Studio

• VS Code/Arduino IDE

• Nordic Semiconductor nRF Cloud

• nRF Bluetooth Low Energy sniffer

• Nordic nRF52840 Dongle

• Fusion 360

• 3D printer

• Qoitech Otii Arc

How it works

Edge Impulse

A model was trained using 394 labeled audio samples of intense coughs, a total of 2 minutes and 34 seconds. An almost equal amount of audio samples of less intense coughs, sneezes, clearing of throat, speech and general sounds was also labeled, 253 samples, 2 minutes and 38 seconds. All samples were captured using the Arduino Nano, positioned at the intended spot for wear.

My coughs lasts around 200 milliseconds. I sampled 10 seconds of repeated coughing with short pauses, then split and trimmed the samples to remove silence.

The model was tested using data set aside and yielded great results. I used EON Tuner in Edge Impulse Studio to find optimal parameters for accuracy, performance and memory consumption.

An Arduino compatible library was built and used to perform continuous interference on an Arduino Nano 33 BLE Sense audio input.

Programming Arduino application

I followed some samples on how to use the generated Arduino libraries from Edge Impulse and how to perform inference on the audio input. If attempting to build my source code, make sure to include the /lib folder. I had to experiment a bit with parameters on the length of the audio window and slices. As each audio sample might start and end in any number of places for a given cough, each piece of audio is analyzed several times, preceeding and following adjoining samples. The results of the inference, the classification, is checked and triggers cough increment if probability is above 50%. A LED is flashed as an indicator.

In short my application defines a custom BLE service, with a characteristic of type unsigned integer, with behaviors Notify, Read and Broadcast. Not very sophisticated, but enough for demonstration. Any connected device will be able to subscribe to updates on the value.

Nordic nRF Cloud

Next I used the nRF Android app on my phone as a gateway between the device and nRF Cloud.

Energy consumption profiling

Energy source

I used lithium polymer batteries for compact size and ease of recharging. I only had spare 500 mAh batteries available, shipping options for assorted batteries by air is limited. To extend battery life I connected two in parallel by soldering 3 JSH female connectors. Warning: This wiring is subjectable to short circuit and is only connected under supervision. This gives twice the capacity while keeping the voltage at the same level.

Electronics work

I made the mistake of assuming I would have to connect more components to the Arduino Nano via a protoboard. On a whim I ordered the Nano with pre-soldered headers. This only took up space and I had to undergo the tedious work of removing the headers by hand using a regular soldering iron. Sacrificing the headers by snipping them every other pin greatly eased the required finger acrobatics.

The only other thing I did was solder a female JSH battery connector to pins VIN and GND. This would serve as my battery connection, and subsequently the device's on/off toggle.

Physical prototype

I wanted to make a prototype for demonstrating the concept for clinicians. It needed to contain and protect the electronics and batteries, while allowing sound waves to reach the microphone. I realized this would complicate making the enclosure watertight and quickly crossed that off the list. I also wanted to make a practical mechanism for securing the device on the wearer.

I used Autodesk Fusion 360 as CAD to design the enclosure. I always start by making rough digital replications of the hardware using calipers to take measurements on a sketch.

This gives the driving constraints and allows me to experiment with different hardware layouts without having to totally scrap alternatives.

While designing I constantly need to take into consideration the manufacturing method, in this case a resin-based SLA 3D printer. When drawing I have to decide on the orientation of the model during printing to avoid complicating overhangs, cupping, hard-to-reach surfaces for removing support material. I also want to reduce the number of parts, to avoid unnecessary post-print work and bad fits.

Completed prints undergo an IPA wash to remove excess resin and finally post cure in a UV-light chamber. What remains is to snip off support material, sand any uneven surfaces and glue together parts. Now the device could finally be assembled and tested.

I ended up with a sort of a badge with a clip and a friction fit lid. It reminds me of a 1960's Star Trek communicator, not the worst thing to be compared against.

Limitations

Battery life is limited to a few days. I am in the process of reimplementing the device using a Neural Decision Processor, NDP, that is able to perform the inference with a fraction of the energy a conventional MCU requires.

I tried to limit audio inference to only perform when the Arduino Nano accelerometer triggers due to some amount of movement (chest movement during a cough). I was disappointed to discover that the interrupt pin on the LSM9DS1 IMU is not connected to the MCU.

You might also have realized that the device will pick up coughs by bystanders, something I discovered when demonstrating the device to a large audience during a conference! Limiting activation to both movement and audio will sort this out.

Future improvements

When demonstrating the device to doctors and nurses I received a great suggestion. A COPD patient that stops taking their daily walk is a great source for concern. My device could be extended to perform monitoring of physical activity using accelerometer data and report aggregated daily activity.

It might be useful to support simple keywords so a patient could log events such as blood in cough, types of pain, self-medication etc.

I plan to move from BLE to LoRaWAN or NB-IoT for transmissions. This way patients won't have to worry about IT administration or infrastructure, it will just work. Please see my other projects at Hackster and element 14 for demonstrations of these lpwan technologies.

Reception

I have had the opportunity to demonstrate the device to clinicians both in person and at expositions and it has received praise, suggestions for further features and use in additional conditions. This project has also spawned several other ideas for wearables in e-health.

Disclosure

Created By: Swapnil Verma

Public Project Link:

Project Demo

Story

Problem

Many times in an application, we want to identify the environment of a user, in order to automatically carry out certain tasks. For example, automatically switching on/off ANC (Automatic Noise Cancellation) in a headphone based on the user's environment. Or, in some applications leaving GPS turned on might be overkill and drain a battery's charge.

In situation's like this, can we detect a user's local environment without the use of GPS?

A Solution

We, humans, are good at understanding our environment just by using auditory sensory skills. I can identify a multitude of sound sources and also guess an environment just by listening. In this project, I am trying to replicate the same behaviour using a TinyML board with a microphone, running a machine-learning model.

Hardware

Training Pipeline

As you might have guessed already, I am using Edge Impulse for building a TinyML model. Let's explore our training pipeline.

1. Data Collection

A good machine learning model starts with a high-quality dataset. I am using the [ESC-50] public dataset, to prepare my own dataset.

I have prepared a total of 7 classes denoting various locations, by combining some of the classes from the ESC-50 dataset. The classes I have prepared are:

• Airport (aeroplane and helicopter sound)

• Bathroom (brushing and toilet flush sound)

• Construction (jackhammer sound)

• Home (washing machine and vacuum cleaner sound)

• Road (siren, car horn and engine sound)

• Work (mouse click and keyboard typing sound)

• Anomaly (quiet environment sound)

I have used only the ESC-50 dataset to prepare a new dataset for this project. The sound samples contained within any of these classes are not the only representation of that class. They can be improved by adding more sounds from different sources.

2. Impulse Design and Preprocessing

The first thing to do for training an ML model is Impulse design. It can also be thought of as a pipeline design from preprocessing to training.

In the pre-processing block, start with the default parameters. If required then change the parameters to suit your needs. In this project, the default parameters worked perfectly, so I just used them.

After adjusting the parameters, click on Generate features.

3. Model Training

My model, trained with 1000 epochs and 0.0002 learning rate has 89.2% accuracy, which is not bad. This tab also shows the confusion matrix, which is one of the most useful tools to evaluate a model. This confusion matrix shows that the work class is the worst-performing class in our dataset.

The pre-processing and NN classifier tab automatically adjusted itself for 6 classes instead, after the dataset was modified.

4. Model Retraining

This will re-run the pre-processing block and learning block in one click, with the latest parameters.

After retraining the model with these parameters, the training accuracy is now 92.1%.

5. Model Testing

Thee Road class has the worst performance. Let's investigate why. Scroll to a sample which was classified incorrectly, and click on the 3 dots to get a menu, and then click on Show classification.

This will show the result of each window of that sample. In the Raw data section, we can also play the audio corresponding to that window, giving us more insight into why the sample was incorrectly classified.

6. Deployment

Live Testing

Let's test the model after deploying it on the Syntiant TinyML board. You can check it out in the below video.

3D-Printable Case

To download a protective 3D-printable case for the Syntiant TinyML board, please follow the below link:

Edge Impulse Project

To clone this Edge Impulse project please follow the below link:

Created By:

Public Project Link:

Project Demo

Project Repo

Introduction

When hospitals are busy it may not always be possible for staff to be close when help is needed, especially if the hospital is short staffed. To ensure that patients are looked after promptly, hospital staff need a way to be alerted when a patient is in discomfort or needs attention from a doctor or nurse.

Solution

A well known field of Artificial Intelligence is voice recognition. These machine learning and deep learning models are trained to recognize phrases or keywords, and combined with the Internet of Things can create fully autonomous systems that require no human interaction to operate.

As technology has advanced, it is now possible to run voice recognition solutions on low cost, resource constrained devices. This not only reduces costs considerably, but also opens up more possibilities for innovation. The purpose of this project is to show how a machine learning model can be deployed to a low cost IoT device (Arduino Nano 33 BLE SENSE), and used to notify staff when a patient needs their help.

The device will be able to detect three keywords Doctor, Nurse, and Help. The device also acts as a BLE peripheral, BLE centrals/masters such as a central server for example, could connect and listen for data coming from the device. The server could then process the incoming data and send a message to hospital staff or sound an alarm.

Hardware

Platform

Software

Project Setup

Create New Project

Your first step is to create a new project. From the project selection/creation you can create a new project.

Enter a project name, select Developer and click Create new project.

We are going to be creating a voice recognition system, so now we need to select Audio as the project type.

Connect Your Device

Once the dependencies are installed, connect your device to your computer and press the RESET button twice to enter into bootloader mode, the yellow LED should now be pulsating.

Once the firmware has been flashed you should see the output above, hit enter to close command prompt/terminal.

Open a new command prompt/terminal, and enter the following command:

edge-impulse-daemon

If you are already connected to an Edge Impulse project, use the following command:

edge-impulse-daemon --clean

Follow the instructions to log in to your Edge Impulse account.

Once complete head over to the devices tab of your project and you should see the connected device.

Data Acquisition

We are going to create our own dataset, using the built in microphone on the Arduino Nano 33 BLE Sense. We are going to collect data that will allow us to train a machine learning model that can detect the words/phrases Doctor, Nurse, and Help.

We will use the Record new data feature on Edge Impulse to record 15 sets of 10 utterances of each of our keywords, and then we will split them into individual samples.

Ensuring your device is connected to the Edge Impulse platform, head over to the Data Acquisition tab to continue.

In the Record new data, make sure you have selected your Arduino Nano 33 BLE Sense, then select Built in microphone, set the label as Doctor, change the sample length to 20000 (20 seconds), and leave all the other settings as.

Here we are going to record the data for the word Doctor. Make sure the microphone is close to you, click Start sampling and record yourself saying Doctor ten times.

You will now see the uploaded data in the Collected data window, next we need to split the data into ten individual samples.

Click on the dots to the right of the sample and click on Split sample, this will bring up the sample split tool. Here you can move the windows until each of your samples are safely in a window. You can fine tune the splits by dragging the windows until you are happy, then click on Split

You will see all of your samples now populated in the Collected data window. Now you need to repeat this action 14 more times for the Doctor class, resulting in 150 samples for the Doctor class. Once you have finished, repeat this for the remaining classes: Nurse and Help. You will end up with a dataset of 450 samples, 150 per class.

Now we have all of our main classes complete, but we still need a little more data. We need a Noise class that will help our model determine when nothing is being said, and we need an Unknown class, for things that our model may come up against that are not in the dataset.

For the noise class we will mix silent samples, and some other general noise samples. First of all record 100 samples with no speaking and store them in an Noise class.

Split Dataset

We need to split the dataset into test and training samples. To do this head to the dashboard and scroll to the bottom of the page, then click on the Perform train/test split

Once you have done this, head back to the data acquisition tab and you will see that your data has been split.

Create Impulse

Now we are going to create our network and train our model.

Head to the Create Impulse tab and change the window size to 2000ms. Next click Add processing block and select Audio (MFCC), then click Add learning block and select Classification (Keras).

Now click Save impulse.

MFCC Block

Parameters

Head over to the MFCC tab and click on the Save parameters button to save the MFCC block parameters.

Generate Features

If you are not automatically redirected to the Generate features tab, click on the MFCC tab and then click on Generate features and finally click on the Generate features button.

Your data should be nicely clustered and there should be as little mixing of the classes as possible. You should inspect the clusters and look for any data that is clustered incorrectly (You don't need to worry so much about the noise and unknown classes being mixed). If you find any data out of place, you can relabel or remove it. If you make any changes click Generate features again.

Training

Now we are going to train our model. Click on the NN CLassifier tab then click Auto-balance dataset, Data augmentation and then Start training.

Once training has completed, you will see the results displayed at the bottom of the page. Here we see that we have 99.2% accuracy. Lets test our model and see how it works on our test data.

Testing

Platform Testing

Head over to the Model testing tab where you will see all of the unseen test data available. Click on the Classify all and sit back as we test our model.

You will see the output of the testing in the output window, and once testing is complete you will see the results. In our case we can see that we have achieved 96.62% accuracy on the unseen data, and a high F-Score on all classes.

On Device Testing

Now we need to test how the model works on our device. Use the Live classification feature to record some samples for classification. Your model should correctly identify the class for each sample.

Performance Calibration

Edge Impulse has a great new feature called Performance Calibration, or PerfCal. This feature allows you to run a test on your model and see how well it will perform in the real world. The system will create a set of post processing configurations for you to choose from. These configurations help to minimize either false activations or false rejections

Once you turn on perfcal, you will see a new tab in the menu called Performance calibration. Navigate to the perfcal page and you will be met with some configuration options.

Select the Noise class from the drop down, and check the Unknown class in the list of classes below, then click Run test and wait for the test to complete.

The system will provide a number of configs for you to choose from. Choose the one that best suits your needs and click Save selected config. This config will be deployed to your device once you download and install the library on your device.

Versioning

We can use the versioning feature to save a copy of the existing network. To do so head over to the Versioning tab and click on the Create first version button.

This will create a snapshot of your existing model that we can come back to at any time.

Deployment

Now we will deploy an Arduino library to our device that will allow us to run the model directly on our Arduino Nano 33 BLE Sense.

Head to the deployment tab and select Arduino Library then scroll to the bottom and click Build.

Note that the EON Compiler is selected by default which will reduce the amount of memory required for our model.

Once the library is built, you will be able to download it to a location of your choice.

Arduino IDE

Once you have downloaded the library, open up Arduino IDE, click Sketch -> Include library -> Upload .ZIP library..., navigate to the location of your library, upload it and then restart the IDE.

Non-Continuous Classification

Open the IDE again and go to File -> Examples, scroll to the bottom of the list, go to AI_Patient_Assistance_inferencing -> nano_ble33_sense -> nano_ble33_sense_microphone.

Once the script is uploaded, open up serial monitor and you will see the output from the program. The green LED on your device will turn on when it is recording, and off when recording has ended.

Now you can test your program by saying any of the keywords when the green light is on. If a keyword is detected the red LED will turn on.

Continuous Classification

Now open AI_Patient_Assistance_inferencing -> nano_ble33_sense -> nano_ble33_sense_microphone_continuous, copy the contents of libraries/ai_patient_assistance/ai_patient_assistance_continuous.ino into the file and upload to your board.

Once the script is uploaded, open up serial monitor and you will see the output from the program. The red LED will blink when a classification is made.

BLE

This program acts as a BLE peripheral which basically advertises itself and waits for a central to connect to it before pushing data to it. In this case our central/master is a smart phone, but in the real world this would be a BLE enabled server that would be able to interact with a database, send SMS, or forward messages to other devices/applications using a machine to machine communication protocol such as MQTT.

When your BLE app connects to the program, the LED light will turn blue, once the app disconnects the LED will turn off.

Conclusion

Here we have created a simple but effective solution for detecting specific keywords that can be part of a larger automated patient assistance system. Using a fairly small dataset we have shown how the Edge Impulse platform is a useful tool in quickly creating and deploying deep learning models on edge devices.

You can train a network with your own keywords, or build off the model and training data provided in this tutorial. Ways to further improve the existing model could be:

• Record more samples for training

• Record samples from multiple people

Created By: Solomon Githu

Public Project Link: https://studio.edgeimpulse.com/public/171255/latest

Overview

In Part 1 of this project, I demonstrated why wearable technology is on the rise and how TinyML is helping to advance the industry. I used the Syntiant TinyML board to run a Machine Learning model that monitors environmental sounds, and predicts when vehicle sounds are detected. Once the device detects the sound of a vehicle or its horn, it generates a vibration, similar to smartphones, that can be perceived by the wearer.

This project can be used, for example, to help people with hearing impairments navigate the streets safely. At the same time, this wearable device is ideal for people strolling through the streets with headphones on, listening to music or podcasts, while inadvertently neglecting the surrounding traffic!

Based on the outcomes of the initial undertaking, I decided to create a wearable device intended for hand placement. This wearable incorporates a protective casing that encompasses all the electronic components, along with wrist straps that allow it to be comfortably worn around the wrist. For the software components, I was satisfied with the model results so I reused the previous Edge Impulse project. For a detailed description on how the model was trained and deployed, please check Part 1 of the project's documentation.

This "Part 2" documentation will outline the process of constructing a similar wearable device. It will provide detailed instructions on how to assemble the device using the publicly available components.

Components and Hardware Configuration

Software components:

• Edge Impulse Studio account

• Edge Impulse Project

• Arduino IDE

Hardware components:

• 3D printed components for the wearable

• Syntiant TinyML board

• Vibration motor module

• 3.7V LiPo battery. I used one with 500mAh

• Veroboard/stripboard

• 1 220 Ω resistor

• 1 2N3904 transistor

• 1 5.7 k Ω resistor

• Some jumper wires and male header pins

• Soldering iron and soldering wire

• Super glue. Always be careful when handling glues!

1. Deploy the Model to the Syntiant TinyML Board

In the Edge Impulse Studio, the Impulse was deployed as an optimized Syntiant NDP 101/120 library. This packages all the signal processing blocks, configuration and learning blocks up into a single package. Afterwards, a custom Arduino code was created to analyze the model's predictions. This code can be obtained from this GitHub repository.

The Arduino code turns GPIO 1 HIGH when an ambulance, firetruck or car siren/horn sounds are detected. GPIO 1 is then used to trigger a motor control circuit that creates a vibration. If you want to turn GPIO 2 or 3 high and low you can use the commands OUT_2_HIGH(), OUT_2_LOW(), OUT_3_HIGH() and OUT_3_LOW() respectively. These functions can be found in the syntiant.h file.

Once the code is uploaded to the Syntiant TinyML board, we can use the Serial Monitor (or any other similar software) to see the logs being generated by the board. Open a terminal, select the COM Port of the Syntiant TinyML board with 115200 8-N-1 settings (in Arduino IDE, that is 115200 baud Carriage return). Sounds of ambulance sirens, firetruck sirens, and cars horns will turn the RGB LED red. For the "unknown" sounds, the RGB LED is off.

2. 3D Print the Wearable Parts

The wearable's components can be categorized into two parts: the electronic components and the 3D printed components.

The 3D printed component files can be downloaded from printables.com or thingiverse.com. The wearable casing is made up of two components: one holds the electrical components while the other is a cover. I 3D printed these with PLA material.

The other 3D printed components are flexible wrist straps. These are similar to the ones found on watches. I achieved the flexibility by printing them with TPU material. Note that if you do not have a good 3D printer you may need to widen the strap's holes after printing.

I then used super glue to attach the wrist straps to the case. Always be careful when handling glues!

Finally, the last 3D printed component is the wearable's dock/stand. This component is not important to the wearable's functionality. A device's dock/stand is just always cool! It keeps your device in place, adds style to your space, and saves you from the existential dread of your device being tangled in cables.

3. Assemble the Electronic Parts

The wearable's electronic components include:

• Syntiant TinyML board

• 3.7V LiPo battery - the wearable's case can hold a LiPo battery which has a maximum dimension of 38mm x 30mm x 5mm

• Vibration motor module

• Circuit board for controlling the vibration motor module - the wearable's case can hold a circuit board that has a maximum dimension of 34mm x 28mm x 5mm

The Syntiant TinyML board has a LiPo battery connector and copper pads that we can use to connect our battery to the board. I chose to solder some wires on the copper pads for the battery connection. Note that the "VBAT" copper pad is for the battery's positive terminal.

The next task is to setup the circuit board for controlling the vibration motor. This control circuit receives a signal from the Syntiant TinyML board GPIO and generates a signal that turns the vibration motor on or off. It can be easily soldered on a veroboard/stripboard with the following components:

• 1 220 Ω resistor – one end connects to the Syntiant GPIO 1, the other ends connects to base of transistor

• 1 2N3904 transistor - the emitter pin is connected to negative terminal of the battery

• 1 5.7 k Ω resistor – one end connects to the positive terminal of the battery, the other end connects to the collector of the transistor

Below is a circuit layout for the wearable. The motor's control circuit is represented by the transistor and resistors on the breadboard. The slide switch is optional. The case however has a slot on the side for placing one.

Once the electronic parts have been assembled, they can be put in the wearable's case according to the layout in the image below.

Below are some images of the assembly that I obtained.

Results

As I was working on the electronic components, I was not so sure if the vibrations from the motor will be noticeable on the wearable. Fortunately, the motor module works very well! The wearable's vibration strength is similar to a smartphone's vibration. This can be seen in the video below showing how the motor vibrates from test code running on the Syntiant TinyML board.

Below is video of the wearable on a person's hand. The Syntiant TinyML board detects an ambulance siren sound in the background and signals it to the person.

Below are some additional images of the wearable.

Conclusion

This environmental sensing wearable is one of the many solutions that TinyML offers. The Syntiant TinyML Board is a tiny board with a microphone and accelerometer, USB host microcontroller and an always-on Neural Decision Processor, featuring ultra low-power consumption, a fully connected neural network architecture, and fully supported by Edge Impulse. I have always been fascinated by this tiny board and this made it the perfect choice for this project!

RatPack is another fascinating wearable that has been created using the Syntiant TinyML board. The huge African pouched rat has been given this gear to enable them to communicate with their human handlers when they come across a landmine or other interesting object. Please checkout the documentation to learn more about this fascinating project. All of the wearable's required components are open source, and the documentation provides step-by-step instructions so you can easily create your own.

Created By: Nekhil R.

Public Project Link: https://studio.edgeimpulse.com/public/170378/latest

Story

In any industry with safety requirements, workers need to be able to report accidents that occur in the workplace. In this project, we are developing a device where employees can simply speak into a microphone to report an accident. The device will be loaded with a machine-learning model that can recognize an accident-reporting keyword. This “keyword spotting” implementation can speed up the reporting process, or make it easier for employees who may not be able to write due to work conditions or literacy. We are only focused on the capability of the device to learn and then recognize our selected keyword here, but future work could include the ability to log events, record audio reports of the incident that has occurred, or other applications.

Hardware

The development board used in this project is the Nordic Thingy:53. It allows you to create prototypes and proof of concept devices without building customized hardware. Its twin Arm Cortex-M33 processors provide enough processing power and memory to execute embedded machine learning (ML) models directly on the development platform.

The Nordic Thingy:53 can be linked to an Edge Impulse Studio account using the nRF Edge Impulse app. It enables you to download trained ML models to the Thingy:53 for deployment and local inferencing, and wirelessly communicate sensor data to a mobile device through Bluetooth LE. It also supports a wired serial connection for to a computer, for use in Edge Impulse Studio. To learn more about configuring the Thingy:53, you can check out the Docs here.

Data Collection

Data Collection is one of the most important steps in a machine learning project. In this project, we used the nRF Edge Impulse app for the Data collection, making it easy to capture input data. The main keyword used here for reporting an accident, is literally the word Accident itself. Below, you can see our data collection settings. We recorded 10 seconds of data, at 16Khz frequency.

This is our keyword collected, along with some noise.

We were able to omit the noise by splitting the sample. So we got our exact keyword, only the piece that we wanted. To make the machine learning model more robust, we collected this same data (the word Accident) from more people. Different ages, different genders, and different talking speeds help make the data better.

In addition to the keyword we'll also need audio that is not our keyword. Background noise, a television ('noise' class), and humans saying other words ('unknown' class) help provide data for the other classes. A machine learning model requires a certain level of "uniformity" in the data it is exposed to, as otherwise it will not be able to learn effectively. The more diverse your data is, the better your machine learning algorithm will perform. So for the Unknown and Noise classes we also made use of this Edge Impulse dataset. To add this data to our project we used the direct Upload method by browsing from the local computer. More information on this dataset can be found here.

We ended up with about 24 minutes of data, which is split it between the Training and Testing:

19m 35s data for Training.

4m 47s data for Testing.

Data Explorer

With the Edge Impulse Data explorer, you can represent your dataset visually, detect outliers, and label unlabeled data. This gives us a quick overview of our complete dataset. To learn more about the Data explorer, you can review the documentation here.

This is our dataset visualization that was generated using the Data explorer.

In this project we only care about the “Accident” keyword, and “Unknown” and “Noise” are not useful for us. As you can see, some of our “Accident” keywords are located in the “Noise” cluster, and a few of them also reside in the “Unknown” cluster of data. Upon examining the ones residing in the Noise cluster a bit closer, we observed that some of them are noises that came about as a result of improper keyword splitting. The remaining items are audio that is unclear or hard to discern the word being spoken, that's how they ended up in the Noise class. Here is one example we found that is indeed an “Accident” keyword that is misclassified in the Noise cluster.

So we deleted the extreme outlying “Accident” keywords that are misclassified and rebalanced the dataset, making it look much cleaner.

Impulse Design

Now it is time to create an Impulse, which is the machine learning pipeline built by Edge Impulse.

For our Processing block, we used Audio (MFE), which is great for the keyword spotting and worked very well with our data.

This is the MFCC Feature generation details page, presented after clicking Save Impulse.

On the right side, we can see the Mel Cepsteral Coefficients of the data, and on the left side the hyper parameters for generating the Mel Frequency Cepstral Coefficients. We are going to leave the default parameters, because they work very well for keyword spotting.

These are the Features generated for our data, shown in a similar format as the Data explorer, for quick visual reference.

With all our data processed, it's time to start training a neural network.

Neural Network

To build our neural network, we simply start by clicking Classifier on the left. The Neural Network default settings might work ok, but you can fine tune them if you need to. More information on the settings can be found in the documentation located here.

Here are our neural network settings and architecture, which work very well for our data.

By enabling Data augmentation, Edge Impulse will make small changes and variations to the dataset, making it more resilient and improving the overall performance by preventing overfitting. Here we used 1-D convolutional architecture which works great for keyword spotting. Let’s take a look at the resulting performance to validate our mode and ensure it’s doing what we expect.

Our dataset is not really that large, and yet we got 94.4% accuracy, so this is pretty good and we can proceed with this model.

Classifying New Data

The next step in the process is to test the model on some new, unseen data before deploying it in the real world.

When we collected data, we set aside that 4 minutes and 47 seconds of data for Testing, which was not used in the creation and training of the model. Now it can be used for Testing, by simply clicking on Model Testing on the left.

After letting it run, it looks like we got around 87% accuracy. Not as high as our predicted 94% accuracy, but still very good.

Deployment

To deploy our model onto the device, back in the mobile App you can simply go to the Deployment tab and click on the Build option. The App will start building our project, and then deploy the firmware directly to our Thingy:53 upon completion.

Real-World Inferencing

After successfully deploying the model to the device, we can start our inferencing by switching to the Inferencing tab in the App.

This video shows real-time inferencing in the Thingy:53.

Conclusion

This project showed how to teach a Thingy:53 to recognize a keyword, commonly known as “keyword spotting”. The process began with recording the word we are interested in recognizing, ensuring we have enough data for a quality dataset, building an Edge Impulse machine learning model, then deploying the model to the device. The keyword used in this project was “Accident”, though any word could be used. Further work on this topic could then add new capabilities beyond the keyword spotting to add voice recording, logging to a dashboard, or similar expanded functionality.

Created By: Naveen Kumar

Public Project Link: https://studio.edgeimpulse.com/public/226454/latest

Overview

It is estimated that more than half of men and over 40% of women in the United States snore, and up to 27% of children. While snoring can be a harmless, occasional occurrence, it can also indicate a serious underlying sleep-related breathing disorder. Snoring is caused by the vibration of tissues near the airway that flutter and produce noise as we breathe. Snoring often indicates obstructive sleep apnea, a breathing disorder that causes repeated lapses in breath due to a blocked or collapsed airway during sleep. Despite being unaware of their snoring, many people suffer from sleep apnea, leading to under-diagnosis. As part of my project, I have developed a non-invasive, low-powered edge device that monitors snoring and interrupts the user moderately through a haptic feedback mechanism.

Hardware Selection

Our system utilizes the Arduino Nicla Voice, which is designed with the Syntiant NDP120 Neural Decision Processor. This processor allows for embedded machine-learning models to be run directly on the device. Specifically designed for deep learning, including CNNs, RNNs, and fully connected networks, the Syntiant NDP120 is perfect for always-on applications with minimal power consumption. Its slim profile also makes it easily portable, which suits our needs.

There are several onboard sensors available on the Nicla Voice, but for this particular project, we will solely make use of the onboard PDM microphone. We are utilizing an Adafruit DRV2605L haptic motor controller and an ERM vibration motor to gently alert users without being intrusive. The haptic motor driver is connected to the Nicla Voice using an Eslov connector and communicates over I2C protocol. The haptic motor driver gets power from the VIN pin since the Eslov connector does not provide power.

Setup Development Environment

To set this device up in Edge Impulse, we will need to install two command-line interfaces by following the instructions provided at the links below.

• Arduino CLI

• Edge Impulse CLI

Please clone the Edge Impulse firmware for this specific development board.

$ git clone https://github.com/edgeimpulse/firmware-arduino-nicla-voice.git

To obtain audio firmware for Nicla Voice, kindly download it from the provided link:

https://cdn.edgeimpulse.com/firmware/arduino-nicla-voice-firmware.zip

To install the Arduino Core for the Nicla board and the pyserial package required to update the NDP120 chip, execute the commands below.

$ unzip arduino-nicla-voice-firmware.zip
$ cd arduino-nicla-voice-firmware
$ ./install_lib_mac.command

We will need to register a free account at Edge Impulse to create a new project.

Data Collection

We have used a dataset available at https://www.kaggle.com/datasets/tareqkhanemu/snoring. Within the dataset, there are two separate directories; one for snoring and the other for non-snoring sounds. The first directory includes a total of 500 one-second snoring samples. Among these samples, 363 consist of snoring sounds created by children, adult men, and adult women without any added background noise. The remaining 137 samples include snoring sounds that have been mixed with non-snoring sounds. The second directory contains 500 one-second non-snoring samples. These samples include background noises that are typically present near someone who is snoring. The non-snoring samples are divided into ten categories, each containing 50 samples. The categories are baby crying, clock ticking, door opening and closing, gadget vibration motor, toilet flushing, emergency vehicle sirens, rain and thunderstorm, street car sounds, people talking, and background television news. The audio files are in the 16-bit WAVE audio format, with a 2-channels (stereo) configuration and a sampling rate of 48,000 Hz. However, we require a single-channel (mono) configuration at a sampling rate of 16,000 Hz, so we need to convert it accordingly using FFmpeg.

$ ffmpeg -loglevel quiet -i <input file> -ar 16000 -ac 1 downsampled/snoring/<output file>

The datasets are uploaded to the Edge Impulse Studio using the Edge Impulse Uploader. Please follow the instructions here to install the Edge Impulse CLI tools and execute the commands below.

$ edge-impulse-uploader --category split --label snoring  downsampled/snoring/*.wav
$ edge-impulse-uploader --category split --label z_openset downsampled/non-snoring/*.wav

To ensure accurate prediction, the Syntiant NDP chips necessitate a negative class that should not be predicted. For the datasets without snoring, the z_openset class label is utilized to ensure that it appears last in alphabetical order. By using the commands provided, the datasets are divided into Training and Testing samples. Access to the uploaded datasets can be found on the Data Acquisition page of the Edge Impulse Studio.

Model Training

Go to the Impulse Design > Create Impulse page and click on the Add a processing block and choose Audio (Syntiant) which computes log Mel-filterbank energy features from an audio signal specific to the Syntiant audio front-end. Also, on the same page click on the Add a learning block and choose Classification which learns patterns from data, and can apply these to new data. We opted for a default window size of 968 ms and a window increase of 32 ms, which allows for two complete frames within a second of the sample. Now click on the Save Impulse button.

Now go to the Impulse Design > Syntiant page and choose the Feature extractors parameter as log-bin (NDP120/200). The number of generated features has to be 1600, which corresponds to the Syntiant Neural Network input layer. To generate 1600 features, it is necessary to validate the following equation:

window size = 1000 x (39 x Frame stride + Frame length)

In our particular case, the window size is 968 ms, which is equivalent to 1000 x (39 x 0.024 + 0.032).

Clicking on the Save parameters button redirects us to another page where we should click on Generate Feature button. It usually takes a couple of minutes to complete feature generation. We can see the visualization of the generated features in the Feature Explorer.

Next, go to the Impulse Design > Classifier page and choose the Neural Network settings as shown below.

We have conducted experiments and established the model architecture using a 2D CNN, which is outlined below.

To begin the training process, click on the Start training button and patiently wait for it to finish. Once completed, we can review the output of the training and the confusion matrix provided below. It's worth noting that the model has achieved an impressive accuracy rate of 99.4%.

Model Testing

To assess the model's performance on the test datasets, navigate to the Model testing page and select the Classify all button. We can be confident in the model's effectiveness on new, unseen data, with an accuracy rate of 94.62%.

Model Deployment

When deploying the model on Nicla Voice, we must choose the Syntiant NDP120 library option on the Deployment page.

To build the firmware, it is necessary to configure the posterior parameters of the Syntiant NDP120 Neural Decision Processor by clicking the Find posterior parameters button.

To detect snoring events, we'll choose that class and disregard z_openset. Also, we are not using any calibration dataset. Then, we'll click Find Parameters to adjust the neural network's precision and recall and minimize the False Rejection Rate and False Activation Rate.

Now click on the Build button and in a few seconds the library bundle will be downloaded to the local computer. We can see the following file listings after unzipping the compressed bundle.

snp-syntiant-ndp120-lib-v1
├── ei_model.synpkg
└── model-parameters
    ├── model_metadata.h
    └── model_variables.h

Now copy the model-parameters content into the src/model-parameters/ directory of the firmware source code that we cloned in the Setup Development Environment section.

$ cp -r snp-syntiant-ndp120-lib-v12/model-parameters/*.h firmware-arduino-nicla-voice/src/model-parameters

We need to customize the firmware code in the firmware-arduino-nicla-voice/src/ei_syntiant_ndp120.cpp file. Add the following declarations near the top section of the file.

#define LAST_N_PREDICTIONS 10
// Haptic motor driver
Adafruit_DRV2605 drv;
// Buffer to keep the last 10 predictions
uint8_t pred_buf[LAST_N_PREDICTIONS] = {0};

In the ei_setup() function, add the haptic feedback motor driver initialization code.

 ...   
    if (! drv.begin()) {
          ei_printf("Could not find DRV2605\n");
          while (true) { delay(10); }
    }
    drv.selectLibrary(1);
    drv.setMode(DRV2605_MODE_INTTRIG);
 ...

You can add custom logic to the main Arduino sketch by customizing the match_event() function as follows.

static void match_event(char* label)
{
    if (_on_match_enabled == true){
        if (strlen(label) > 0) {
            got_match = true;
            ei_printf("Match: %s\n", label); 
            if (strcmp(label, "NN0:snoring") == 0) {
                nicla::leds.setColor(green);
                delay(500);
 
                memmove(pred_buf, pred_buf+1, LAST_N_PREDICTIONS-1);            
                pred_buf[LAST_N_PREDICTIONS-1] = 1;
 
                int sum = 0;
                for (int i = 0; i < LAST_N_PREDICTIONS; i++) {
                    sum += pred_buf[i];
                    ei_printf("%d ", pred_buf[i]);
                }
                ei_printf("\n");
                
                if (sum >= 3) {
                    ei_printf("Snoring Detected");
                    // reset buffer
                    for (int i = 0; i < LAST_N_PREDICTIONS; i++) {
                       pred_buf[i] = 0;
                    }

                    int count = 5;
                    while (count > 0) {
                        drv.setWaveform(0, 14); // play effect strong buzz
                        drv.setWaveform(1, 0);  // end waveform
                        drv.go();
                        delay(500);
                        count--;
                    }
                }
            }

            if (strcmp(label, "NN0:unknown_noise") == 0) {
                nicla::leds.setColor(red);
                // rotate the buffer by 1 element
                memmove(pred_buf, pred_buf+1, LAST_N_PREDICTIONS-1);
                pred_buf[LAST_N_PREDICTIONS-1] = 0;

                for (int i = 0; i < LAST_N_PREDICTIONS; i++) {
                    ei_printf("%d ", pred_buf[i]);
                }
                ei_printf("\n");
            }
        }
    }
}

To compile the firmware execute the command below.

$ cd firmware-arduino-nicla-voice
$ ./arduino-build.sh --build

Once we've compiled the Arduino firmware we need to follow the steps below to deploy the firmware to the device. It is assumed that all firmware and deployment libraries are in the same directory level.

• Take the .elf file output by Arduino and rename it to firmware.ino.elf.

• Replace the firmware.ino.elf from the default firmware

• Replace the ei_model.synpkg file in the default firmware by the one from the downloaded Syntiant library.

$ mv firmware-arduino-nicla-voice.ino.elf firmware.ino.elf
$ cd ../arduino-nicla-voice-firmware
$ cp ../firmware-arduino-nicla-voice/firmware.ino.elf .
$ cp ../snp-syntiant-ndp120-lib-v1/ei_model.synpkg ndp120/

Now connect the Nicla Voice to the computer using a USB cable and execute the script (OS-specific) to flash the board.

$ ./flash_mac.command 

We can monitor the inferencing results over the serial connection at a 115200 baud rate after resetting the board or power cycling.

dsp firmware version: 19.2.0
package version: Model generated by Edge Impulse
pbi version: 3.2.3-0
num of labels: 2
labels: NN0:snoring, NN0:z_openset
total deployed neural networks: 1
IMU not enabled
Inferencing settings:
.Interval: 0.062500 ms.
.Frame size: 15488
.Sample length: 968 ms.
.No. of classes: 2
Classes:
.snoring
.z_openset
Starting inferencing, press 'b' to break
Type AT+HELP to see a list of commands.
> 
Match: NN0:snoring
0 0 0 0 0 0 0 0 0 1 
Match: NN0:snoring
0 0 0 0 0 0 0 0 1 1 

Inferencing Demo

When classifying audio, it is recommended to analyze multiple windows of data and calculate the average of the results. This approach can provide better accuracy compared to relying on individual results alone. To detect snoring events, the application maintains a circular buffer of the last 10 inferencing results. If three snoring events are detected, the haptic feedback motor will vibrate for a few seconds. During the demo, the motor vibration appears to be moderate. However, the user can still feel the vibration while using the pillow.

Conclusion

This project provides a practical solution to a seemingly humorous yet serious issue. The device is user-friendly and respects privacy by conducting inferencing at the edge. Although the model performs well, there is potential for enhancement through the inclusion of more refined training data, resulting in improved accuracy and robustness to further reduce the likelihood of false positives and negatives. Additionally, this project showcases the ability of a simple neural network to address complex problems, executed through effective digital signal processing on low-powered, resource-restricted devices such as Syntiant NDP120 Neural Decision Processor on the Nicla Voice.

Created By: Solomon Githu

Public Project Link: https://studio.edgeimpulse.com/public/111611/latest

Industrial Automation Inefficiencies Towards Accident Detection

The International Labor Organization estimates that there are over 1 million work-related fatalities each year and millions of workers suffer from workplace accidents. However, even as technology advancements have improved worker safety in many industries, some accidents involving workers and machines remain undetected as they occur, possibly even leading to fatalities. This is because of the limitations in Machine Safety Systems. Safety sensors, controllers, switches and other machine accessories have been able to provide safety measures during accidents but some events remain undetected by these systems.

Some accidents which are difficult to be detected in industries includes:

• Falling Objects

• Objects strike or fall on employees

• Slips or Falls of employees

• Chemical burns or exposure

• Workers caught in moving machine parts

Detecting Worker Accidents with AI

Sound classification is one of the most widely used applications of Machine Learning. When in danger or scared, we humans respond with audible actions such as screaming, crying, or with words such as: "stop", or "help" . This alerts other people that we are in trouble and can also give them instructions such as stopping a machine, or opening/closing a system. We can use sound classification to give hearing to machines and manufacturing setups so that they can be aware of the environment status.

TinyML has enabled us to bring machine learning models to low-cost and low-power microcontrollers. We will use Edge Impulse to develop a machine learning model which is capable of detecting accidents from workers screams and cries. This event can then be used to trigger safety measures such as machine/actuator stop, and sound alarms.

The Syntiant TinyML Board is a tiny development board with a microphone and accelerometer, USB host microcontroller and an always-on Neural Decision Processor™, featuring ultra low-power consumption, a fully connected neural network architecture, and fully supported by Edge Impulse. Here are quick start tutorials for Windows and Mac.

Quick Start

You can find the public project here: Acoustic Sensing of Worker Accidents. To add this project into your account projects, click "Clone this project" at the top of the window. Next, go to the "Deploying to Syntiant TinyML Board" section below to see how you can deploy the model to the Syntiant TinyML board.

Alternatively, to create a similar project, follow the next steps after creating a new Edge Impulse project.

Data Acquisition

We want to create a model that can recognize both key words and human sounds like cries and screams. For these, we have 4 classes in our model: stop, help, cry and scream. In addition to these classes, we also need another class that is not part of our 4 keywords. We label this class as "unknown" and it has sound of people speaking, machines, and vehicles, among others. Each class has 1 second of audio sounds.

In total, we have 31 minutes of data for training and 8 minutes of data for testing. For the "unknown" class, we can use Edge Impulse Key Spotting Dataset, which can be obtained here. From this dataset we use the "noise" audio files.

Impulse Design

The Impulse design is very unique as we are targeting the Syntiant TinyML board. Under 'Create Impulse' we set the following configurations:

Our window size is 968ms, and window increase is 484ms milliseconds(ms). Click 'Add a processing block' and select Audio (Syntiant). Next, we add a learning block by clicking 'Add a learning block' and select Classification (Keras). Click 'Save Impulse' to use this configuration.

Next we go to our processing block configuration, Syntiant, and first click 'Save parameters'. The preset parameters will work well so we can use them in our case.

On the window 'Generate features', we click the "Generate features" button. Upon completion we see a 3D representation of our dataset. These are the Syntiant blocks that will be passed into the neural network.

Lastly, we need to configure our neural network. Start by clicking "NN Classifier" . Here we set the number of training cycle to 80, with a learning rate of 0.0005. Edge Impulse automatically designs a default Neural Network architecture that works very well without requiring the parameters to be changed. However, if you wish to update some parameters, Data Augmentation can improve your model accuracy. Try adding noise, masking time and frequency bands and asses your model performance with each setting.

With the training cycles and learning rate set, click "Start training", and you will have a neural network when the task is complete. We get an accuracy of 94%, which is pretty good!

Model Testing

When training our model, we used 80% of the data in our dataset. The remaining 20% is used to test the accuracy of the model in classifying unseen data. We need to verify that our model has not overfit by testing it on new data. If your model performs poorly, then it means that it overfit (crammed your dataset). This can be resolved by adding more data and/or reconfiguring the processing and learning blocks if needed. Increasing performance tricks can be found in this guide.

On the left bar, click "Model testing" then "classify all". Our current model has a performance of 91% which is pretty good and acceptable.

From the results we can see new data called "testing" which was obtained from the environment and sent to Edge Impulse. The Expected Outcome column shows which class the collected data belong to. In all cases, our model classifies the sounds correctly as seen in the Result column; it matches the Expected outcome column.

Deploying to the Syntiant TinyML Board

To deploy our model to the Syntiant Board, first click "Deployment". Here, we will first deploy our model as a firmware on the board. When our audible events (cry, scream, help, stop) are detected, the onboard RGB LED will turn on. When the unknown sounds are detected, the on board RGB LED will be off. This runs locally on the board without requiring an internet connection, and runs with minimal power consumption.

Under "Build Firmware" select Syntiant TinyML.

Next, we need to configure posterior parameters. These are used to tune the precision and recall of our Neural Network activations, to minimize False Rejection Rate and False Activation Rate. More information on posterior parameters can be found here: Responding to your voice - Syntiant - RC Commands, in "Deploying to your device" section.

Under "Configure posterior parameters" click "Find posterior parameters". Check all classes apart from "unknown", and for calibration dataset we use "No calibration (fastest)". After setting the configurations, click "Find parameters".

This will start a new task which we have to wait until it is finished.

When the job is completed, close the popup window and then click "Build" options to build our firmware. The firmware will be downloaded automatically when the build job completes. Once the firmware is downloaded, we first need to unzip it. Connect a Syntiant TinyML board to your computer using a USB cable. Next, open the unzipped folder and run the flashing script based on your Operating System.

We can connect to the board's firmware over Serial. To do this, open a terminal, select the COM Port of the Syntiant TinyML board with settings 115200 8-N-1 settings (in Arduino IDE, that is 115200 baud Carriage return).

Sounds such as "stop", "help", "aaagh!" or crying will turn the RGB LED to red.

For the "unknown" sounds, the RGB LED is off. While configuring the posterior parameters, the detected classes that we selected are the ones which trigger the RGB LED lighting.

Taking it one step further

We can use our Machine Learning model as a safety feature for actuators, machines or other operations involving people and machines.

To do this we can build custom firmware for our Syntiant TinyML board that turns a GPIO pin HIGH or LOW based on the detected event. The GPIO pin can then be connected to a controller that runs an actuator or a system. The controller can then turn off the actuator or process when a signal is sent by the Syntiant TinyML board.

A custom firmware was then created to turn on GPIO 1 HIGH (3.3V) of the Syntiant TinyML Board whenever the alarming sounds are detected. GPIO 1 is next to the GND pin so we can easily use a 2-pin header to connect our TinyML board with another device.

Awesome! What's next now? Checkout the custom firmware here and add intelligent sensing to your actuators and also home automation devices!

Intelligent sensing for 8-bit LoRaWAN actuator

I leveraged my TinyML solution and used it to add more sensing to my LoRaWAN actuator. I connected the Syntiant TinyML board to an Atmega and SX1276 based development board called the WaziAct. This board is designed to play as a production LoRa actuator node with an onboard relay which I often use to actuate pumps, solenoids, and electrical devices. I programmed the board to read the pin status connected to the Syntiant TinyML board and when a signal is received it stops executing the main tasks. An alert is also sent to the gateway via LoRa while the main tasks remain halted. The Arduino code can be accessed here.

Below is a sneak peak of an indoor test… Now my "press a button" LoRaWAN actuations can run without causing harm such as turning on a faulty device, pouring water via solenoid/pump in unsafe conditions, and other accidental events!

Conclusion

We have seen how we can use sounds to train and deploy our ML solution easily and also run them locally on a development board. TinyML-based intelligent sensing, such as is shown here, is just one of the many solutions that TinyML offers.

With Edge Impulse, developing ML models and deploying them has always been easy. The Syntiant TinyML board was chosen for this project because it provides ultra-low power consumption, fully connected neural network architecture, has an onboard microphone, is physically small, and is also fully supported by Edge Impulse.

Created By: Nekhil R.

Public Project Link: https://studio.edgeimpulse.com/public/149095/latest

GitHub Repository: https://github.com/CodersCafeTech/Vandalism-Detection

Project Demo

Story

The direct annual cost of vandalism runs in the billions of dollars annually in the United States alone. Breaking glass and defacing property are some of the serious forms of vandalism. Conventional security techniques such as direct lighting and intruder alarms can be ineffective in so locations and cases, so here we explore another form of prevention. In this project, we are able to detect the sound of glass breaking, and can alert a user instantly about the event.

In this project, we only focus on glass breaking, however, this project can be applied to any other form of vandalism that also produces a unique sound.

How Does It Work

The device will work as follows. Suppose a vandal tried to break glass, which will of course have a unique sound. The tinyML model running on the device can recognize the event using a microphone. Then the device will send email notifications to a registered user regarding the audio detection.

Hardware

Arduino Nano 33 BLE Sense

For this project we are using an Arduino Nano 33 BLE Sense. It's a 3.3V AI-enabled board in a very small form factor. It comes with a series of embedded sensors including an MP34DT05 Digital Microphone.

ESP-01

The ESP-01 is used for adding WiFi capability to the device, because the Arduino Nano 33 BLE Sense does not have any native WiFi capability. The WiFi is specifically used for sending email alerts. Serial communication is established between the Arduino and ESP-01, for transmitting the email.

Software

Data Acquisition

One of the most important parts of any machine learning model is its dataset. Edge Impulse offers us two options to create our dataset: either direct uploading of files, or recording data from actual the device itself. For this project we chose to record data with the device itself, because as a prototype, the data will be limited. A second reason to record data with the device itself is that it can improve accuracy. To get started connecting the Nano 33 BLE Sense to Edge Impulse, you can have a look at this tutorial.

In this scenario, we have only two classes Glass Break, and Noise. Glass breaking sounds that we have used are from the vivid online resources and the major noise datasets are from the Microsoft Scalable Noisy Speech Dataset (MS-SNSD). We also included the natural noise in the room, apart from the MS-SNSD data.

The sound recording was done for 20 seconds at a 16KHz sampling rate. Something to keep in mind is that you must keep the sampling rate the same between your training dataset and your deployment device. If you are training with 44.1Khz sound, you need to downsample it to 16KHz when you are ready to deploy to the Arduino.

We collected around 10 minutes of data and split it between Training and a Test set. In the Training data we split the samples to 2s, otherwise the inferencing will fail because the BLE Sense has a limited amount of memory to handle the data.

Impulse Design

This is our Impulse, which is the machine learning pipeline termed by Edge Impulse.

Here we used MFE as the processing block, because it is very suitable for non-human voices. We have used the default parameters of the MFE block.

Neural Network

These are our Neural Network settings, which we found most suitable for our data. If you are tinkering with your own dataset, you might need to change these parameters a bit, and some exploration and testing could be required.

We enabled the Data augmentation feature, which helps us to randomly transform data during training. This we are able to run more training cycles without overfitting the data, and also helps improve accuracy.

This is our Neural Network architecture.

We have used the default 1D convolutional layer, then we trained the model. We ended up with 97% accuracy, which is very awesome. By looking at the confusion matrix it is clear that there is no sign of underfitting and overfitting.

Model Testing

Before deploying the model, it's a good practice to run the inference on the Test dataset that was set aside earlier. In the Model Testing, we got around 92% accuracy.

Let's look into some of the misclassifications, to better understand what is happening. In this case, the noise very well resembled the Glass Break sound, which is why it is misclassified:

In this next case, the model performed very well in classifying the data, although the data contains both the Glass_Break and some noise. The majority of the data was noise however, that's why its misclassified.

In these two cases shown below, again Noise was the major reason for misclassification:

Overall though, the model is performing very well and can be deployed in the real world.

Deployment

For deploying the Impulse to the BLE Sense, we exported the model as an Arduino library from the Studio.

Then we add that library to the Arduino IDE. Next, we modified the example sketch that is provided, to build our application. You can find the code and all assets including the circuit in this GitHub Repository.

IFTT

For triggering the email we used the IFTT service. To setup the mail triggering upon any positive detections, please have a look at this tutorial.

Here is the application I have created:

Case

All the components were fit inside this case, to make a tidy package:

Real World Testing

Here are the results of some live testing, after the model is deployed to the device. You can check it out in the below video. The sound of the glass breaking is played on a speaker, and you can see the results of the inferencing and email being sent.

Created By: Shebin Jose Jacob

Public Project Link: https://studio.edgeimpulse.com/public/162492/latest

Intro

Every manufacturing environment is equipped with machines. For a better-performing manufacturing unit, the health of machines plays a major role and hence maintenance of the machines is important. We have three strategies of maintenance namely - Preventive maintenance, Corrective maintenance, and Predictive maintenance.

If you want to find the best balance between preventing failures and avoiding over-maintenance, Predictive Maintenance (PdM) is the way to go. Equip your factory with relatively affordable sensors to track temperature, vibrations, and motion data, use predictive techniques to schedule maintenance when a failure is about to occur, and you'll see a nice reduction in operating costs.

In the newest era of technology, teaching computers to make sense of the acoustic world is now a hot research topic. So in this project, we use sound to do some predictive maintenance using an Arduino Nano 33 BLE Sense.

How Does It Work?

We use the Nano 33 BLE Sense to listen to the machine continuously. The MCU runs an ML model which is trained on two sets of acoustic anomalies and a normal operation mode. When the ML model identifies an anomaly, the operator is immediately notified and the machine may be shut down for maintenance after proper inspection. Thus, we can reduce the possible damage caused and can reduce the downtime.

Hardware Requirements

• Nano 33 BLE Sense

• LED

Software Requirements

• Edge Impulse

• Arduino IDE

Hardware Setup

The hardware setup consists of a Nano 33 BLE Sense, which is placed beside an old AC motor.

Software Setup

If you haven't connected the device to Edge Impulse dashboard, follow this tutorial to get it connected. After a successful connection, it should be present in the Devices tab.

Alternatively, recent versions of Google Chrome and Microsoft Edge can collect data directly from your development board, without the need for the Edge Impulse CLI. Follow this tutorial to learn more about it.

TinyML Model Generation

1. Data Collection

Clean data is the most important requirement to train a well-performing model. In our case, we have collected 3 classes of sound - two classes of anomalies, one normal operation class, and a noise class. Each sample is 2 seconds long. The raw data of these classes is visualised below.

If the data is not split into training and testing datasets, split the dataset into training and testing datasets in the ratio 80:20, which forms a good dataset for model training.

2. Impulse Design

An Impulse is the machine learning pipeline that takes raw data, uses signal processing to extract features, and then uses a learning block to classify new data.

Here we are using the Time Series data as the input block. Now, we have two choices for the processing block - MFCC and MFE. As we are dealing with non-vocal audio and MFE performs well with non-vocal audio, we have chosen MFE as our processing block. We have used Classification as our learning block since we have to learn patterns and apply them to new data to categorize the audio into one of the given 4 classes.

In the MFE tab, you can tweak the parameters if you're good with audio handling, else leave the settings as it is and generate features.

3. Model Training and Testing

Now that we have our Impulse designed, let's proceed to train the model. The settings we employed for model training are depicted in the picture. You can play about with the model training settings so that the trained model exhibits a higher level of accuracy, but be cautious of overfitting.

A whopping 94.7% accuracy is achieved by the trained model.

Let's now use some unknown data to test the model's functionality. To assess the model's performance, move on to Model Testing and Classify All.

We have got 95.07% testing accuracy, which is pretty awesome. Now let's test the model with some real-world data. Navigate to Live Classification and collect some data from the connected device.

We have collected some real-world data of Normal Operation Mode, Anomaly 1, and Anomaly 2 respectively, and all of them are correctly classified. So our model is ready for deployment.

Deployment

For deployment, navigate to the Deployment tab, select Arduino Library and build the library. It will output a zip library, which can be added to Arduino IDE.

Final Product

Nano 33 BLE Sense along with an LED is enclosed in a 3D printed case, which is our final product. The device is capable of identifying acoustic anomalies in a machine and alerts the user using the alert LED.

Created By: Swapnil Verma

Public Project Link: https://studio.edgeimpulse.com/public/133765/latest

Intro Problem

On May 24, 2022, nineteen students and two teachers were fatally shot, and seventeen others were wounded at Robb Elementary School in Uvalde, Texas, United States[1]. An 18-year-old gunman entered the elementary school and started shooting kids and teachers with a semi-automatic rifle. The sad part is that it is not a one-off event. Gun violence including mass shootings is a real problem, especially in the USA.

What Can I Do About It?

Gun violence is a massive problem and I alone can not solve it, but I can definitely contribute an engineering solution toward hopefully minimizing casualties.

Here I am proposing a proof of concept to identify gun sounds using a low-cost system and inform emergency services as soon as possible. Using this system, emergency services can respond to a gun incident as quickly as possible thus hopefully minimizing casualties.

How Does It Work?

My low-cost proof of concept uses multiple microcontroller boards with microphones to capture sound. They use a TinyML algorithm prepared using Edge Impulse to detect any gunshot sound. Upon a positive detection, the system sends a notification to registered services via an MQTT broker.

To learn more about the working of the system, please check out the Software section.

Hardware

The hardware I am using is:

• Arduino Portenta H7

• Arduino Nano BLE Sense

• 9V Battery

In the current hardware iteration, the Arduino Nano BLE Sense is powered by a 9V battery and the Portenta H7 is powered via a laptop, because I am also using the serial port on the Portenta H7 to debug the system.

Software

The software is divided into 2 main modules:

• Machine Learning

• Communication

Machine Learning Module

The machine learning module uses a tinyML algorithm prepared using Edge Impulse. This module is responsible for identifying gunshot sounds. It takes sound as input and outputs its classification.

Dataset

One of the most important parts of any machine learning model is its dataset. In this project, I have used a combination of two different datasets. For the gunshot class, I used the Gunshot audio dataset and for the other class, I used the UrbanSound8K dataset from Kaggle.

Edge Impulse Studio's Data acquisition tab provides useful features to either record your own data or upload already-collected datasets. I used the upload feature.

While uploading you can also provide a label to the data. This was very helpful because I am using sounds from multiple origins in the other class. After uploading the data, I cleaned the data using the Studio's Crop and Split sample feature.

You can either divide data into test and train sets while uploading or do it at a later time. The Data acquisition tab shows the different classes and train/test split ratio for our convenience.

Training

After preparing the dataset, we need to design an Impulse. The Edge Impulse documentation explains the Impulse design in great detail so please check out their documentation page to learn about Impulse design.

As you can see, I have selected MFCC as the preprocessing block, and classification as the learning block. I have used the default parameters for the MFCC preprocessing. For training, I have slightly modified the default neural network architecture. I have used three 1D convolutional CPD layers with 8, 16 and 24 neurons, respectively. The architecture is illustrated in the below image.

Modifying the neural network architecture in the Edge Impulse Studio is very easy. Just click on Add an extra layer at the bottom of the architecture and select any layer from the popup.

Or you can also do it from the Expert mode if you are feeling masochistic 😉.

I trained the model for 5000 iterations with a 0.0005 learning rate. My final model has 94.5% accuracy.

Testing

The Edge Impulse Studio's Model testing tab enables a developer to test the model immediately. It uses the data from the test block and performs the inference using the last trained model. My model had 91.3% accuracy on the test data.

Versioning

One nice feature Edge Impulse provides is versioning. You can version your project (like Git) to store all data, configuration, intermediate results and final models. You can revert back to earlier versions of your impulse by importing a revision into a new project. I use this feature every time before changing the neural network architecture. That way I don't have to retrain or keep a manual record of the previous architecture.

Deployment

After completing the training, it's time for deployment. The Deployment tab of the Edge Impulse Studio provides three main ways of deploying the model onto hardware: (a) by creating a library (b) by building firmware, and (c) by running it on a computer or a mobile phone directly. I knew that I need more functionality from my Arduino hardware apart from inferencing, so I created a library instead of building firmware.

Just select the type of library you want, and click the Build button at the bottom of the page. This will build a library and download it onto your computer. After downloading, it will also show a handy guide to include this library in your IDE.

The coolest part is that I don't need to retrain the model or do anything extra to deploy the same model onto multiple devices. The examples of the downloaded library will have example code for all the supported devices of the same family.

Just select an example as a getting started code, modify it according to your need and flash it. The Arduino Nano BLE Sense and Portenta H7 use the same model generated by Edge Impulse. I trained the model only once, agnostic of the hardware and deployed it on multiple devices which is a time saver.

Inferencing

Inferencing is the process of running a neural network model to generate output. The image below shows the inference pipeline.

The microphones in the Nano BLE Sense and Portenta H7 pick up the surrounding sound (stages 1 & 2). The sound data is then preprocessed using the MFCC block (stage 3). The preprocessed data is then sent to a Convolutional Neural Network block (stage 4) which classifies it into either the gunshot class or the other class (stage 5).

To learn more about the project please follow the below link.

Project Link - https://studio.edgeimpulse.com/public/133765/latest

The output of the machine learning module is then processed before sending it to the cloud via the Communication module.

Communication Module

This module is responsible for sharing information between boards and sending positive predictions to the registered emergency services.

The Nano BLE Sense sends its inference to the Portenta H7 via BLE. The Portenta H7 then sends the positive output (i.e. detection of gunshot sound) of its inference and Nano BLE's inference to a subscriber via MQTT. I have used the cloudMQTT broker for this project.

To download the software please use the below link:

Software Link - https://github.com/sw4p/Gunshot_Detection

Testing

My testing setup and the result are illustrated in the video below. The system is connected to my laptop which is also performing the screen recording. On the upper left side, we have an Arduino serial window which is showing the output from the Portenta H7, and on the lower left hand, we have an audio player. On the right-hand side, we have cloudMQTT's WebSocket UI, which shows the incoming notification via MQTT. The sound for this video is played and recorded using my laptop's speaker and microphone.

In the video above I am playing different categories of sound and one of that categories is a gunshot. The system outputs the classification result to the Arduino serial port whenever it detects a sound from the other class but does not send it to the receiver. The moment it detects a gunshot sound, it immediately sends a notification to the receiver via CloudMQTT.

Code

https://github.com/sw4p/Gunshot_Detection

References

[1] https://en.wikipedia.org/wiki/Robb_Elementary_School_shooting

[2] https://www.gunviolencearchive.org/

Created By: Solomon Githu

Public Project Link: https://studio.edgeimpulse.com/public/171255/latest

Introduction

An electronic device that is intended to function on a user's body is considered wearable technology. The largest categories of wearables are smartwatches and hearables, which have experienced the fastest growth in the recent years. Steve Roddy, former Vice President of Product Marketing for Arm's Machine Learning Group once said that "TinyML deployments are powering a huge growth in ML deployment, greatly accelerating the use of ML in all manner of devices and making those devices better, smarter, and more responsive to human interaction". TinyML enables running Machine Learning on resource constrained devices like wearables.

Sound classification is one of the most widely used applications of Machine Learning. A new use case for wearables is an environmental audio monitor for individuals with hearing disabilities. This is a wearable device that has a computer which can listen to the environment sounds and classify them. In this project, I focused on giving tactile feedback when vehicle sounds are detected. The Machine Learning model can detect ambulance and firetruck sirens as well as cars honking. When these vehicles are detected, the device then gives a vibration pulse which can be felt by the person wearing the device. This use case can be revolutionary for people who have hearing problems and even deaf people. To keep people safe from being injured, the device can inform them when there is a car, ambulance or firetruck nearby so that they can identify it and move out of the way.

I used Edge Impulse Platform to train my model and deploy it to the Syntiant TinyML board. This is a tiny development board with a microphone and accelerometer, USB host microcontroller and an always-on Neural Decision Processor™, featuring ultra low-power consumption, a fully connected neural network architecture, and supported by Edge Impulse.

You can find the public Edge Impulse project here: Environmental Audio Monitor. To add this project into your Account projects, click "Clone" at the top of the window. Next, go to "Deploying to Syntiant TinyML board" section to see how you can deploy the model to the Syntiant TinyML board.

Dataset Preparation

I first searched for open datasets of ambulance siren, firetruck siren, car horns and traffic sounds. I used the Kaggle dataset of Emergency Vehicle Siren Sounds and the Isolated urban sound database for the key sounds. From these datasets, I created the classes "ambulance_firetruck" and "car_horn".

In addition to the key events that I wanted to be detected, I also needed another class that is not part of them. I labelled this class as "unknown" and it has sounds of traffic, people speaking, machines, and vehicles, among others. Each class has 1 second of audio sounds.

In total, I had 20 minutes of data for training and 5 minutes of data for testing. For part of the "unknown" class, I used Edge Impulse keywords dataset. From this dataset, I used the “noise” audio files.

Impulse Design

The Impulse design was very unique as I was targeting the Syntiant TinyML board. Under "Create Impulse" I set the following configurations:

The window size is 968ms and window increase is 484ms milliseconds(ms). I then added the "Audio (Syntiant)" processing block and the "Classification" Learning block. For a detailed explanation of the Impulse Design for the Syntiant TinyML audio classification, checkout the Edge Impulse documentation.

The next step was to extract Features from the training data. This is done by the Syntiant processing block. On the Parameters page, I used the default Log Mel filterbank energy features and they worked very well. The Feature explorer is one of the most fun options in Edge Impulse. In the feature explorer, all data in your dataset are visualized in one graph. The axes are the output of the signal processing process and they can let you quickly validate whether your data separates nicely. I was satisfied with how my features separated for each class. This enabled me to proceed to the next step, training the model.

Under "Classifier" I set the number of training cycle as 100 with a learning rate of 0.0005. Edge Impulse automatically designs a default Neural Network architecture that works very well without requiring the parameters to be changed. However, if you wish to update some parameters, Data Augmentation can improve your model accuracy. Try adding noise, masking time and frequency bands and inspect your model performance with each setting.

I then clicked “Start training” and waited for a few minutes for the training to be complete. Upon completion of the training process, I got an accuracy of 97.6%, which is pretty good!

Model Testing

When training the model, I used 80% of the data in the dataset. The remaining 20% is used to test the accuracy of the model in classifying unseen data. We need to verify that our model has not overfit by testing it on new data. If your model performs poorly, then it means that it overfit (crammed your dataset). This can be resolved by adding more dataset and/or reconfiguring the processing and learning blocks if needed. Increasing performance tricks can be found in this guide.

On the left bar, we click "Model testing" then "Classify all". The current model has a performance of 97.8% which is pretty good and acceptable.

From the test data, we can see the first sample has a length of 3 seconds. I recorded this in a living room which had a computer playing siren sounds and at the same time a television was playing a movie. In each timestamp of 1 second, we can see that the model was able to predict the ambulance_firetruck class. I took this as an acceptable performance of the model and proceeded to deploy it to the Syntiant TinyML board.

Deploying to Syntiant TinyML Board

To deploy our model to the Syntiant Board, first click "Deployment" on the left side panel. Here, we will first deploy our model as a firmware for the board. When our audible events: ambulance_firetruck and car_horn are detected, the onboard RGB LED will turn on. When the "unknown" sounds are detected, the onboard RGB LED will be off. This firmware runs locally on the board without requiring internet connectivity and also with minimal power consumption.

Under "Build Firmware" select Syntiant TinyML.

Next, we need to configure posterior parameters. These are used to tune the precision and recall of our Neural Network activations, to minimize False Rejection Rate and False Activation Rate. More information on posterior parameters can be found here Responding to your voice - Syntiant - RC Commands

Under "Configure posterior parameters" click "Find posterior parameters". Check all classes apart from "unknown", and for calibration dataset we use "No calibration (fastest)". After setting the configurations, click "Find parameters".

This will start a new task, so we have to wait until it is finished.

When the job is completed, close the popup window and then click "Build" option to build our firmware. The firmware will be downloaded automatically when the build job completes.

Once the firmware is downloaded, we first need to unzip it. Connect a Syntiant TinyML board to your computer using a USB cable. Next, open the unzipped folder and run the flashing script based on your Operating System.

We can connect to the board's firmware over Serial. To do this, open a terminal, select the COM Port of the Syntiant TinyML board with settings 115200 8-N-1 settings (in Arduino IDE, that is 115200 baud Carriage return). Sounds of ambulance sirens, firetruck sirens, and cars horns will turn the RGB LED red.

For the "unknown" sounds, the RGB LED is off. In configuring the posterior parameters, the detected classes that we selected are the ones which trigger the RGB LED lighting.

A Smart Watch-out

After testing the model on the Syntiant TinyML board and finding that it works great, I proceeded to create a demo of the smart wearable of this project.

This involved connecting a vibration motor to GPIO 1 of the Syntiant TinyML board. When the classes "ambulance_firetruck" and "car_horn" are detected, the GPIO 1 on the board is set HIGH and this causes the vibration motor to vibrate for 1500 milliseconds. Vibration motors are mostly used to give haptic feedback in mobile phones and video game controllers. They are the components that make your phone vibrate.

Since we cannot connect a motor directly to the GPIO pins, I used the 5V pad on the Syntiant TinyML board to power the vibration motor through a transistor that is switched by GPIO 1.

In future, we can then package these components safely into a wrist wearable. The Syntiant TinyML board has a 3.7V LiPo battery connector which will enable the wearable to be used anywhere. For this demo, I used the USB connector as the power source for all components.

You can find the Arduino code for this use case in the GitHub repository syntiant-tinyml-firmware-environment-hearing-aider. The repository has the instructions on how to install the required libraries and upload the Arduino code to the Syntiant TinyML board.

The image below shows the annotation of the Syntiant TinyML board. GPIO 1, GND and the 5V pad on the bottom side are used for this smart wearable.

Conclusion

This environmental audio monitor wearable is one of the many solutions that TinyML offers. A future work can be to include other sounds such as motor bikes, detect construction equipment, or alarm sounds, among others.

With Edge Impulse, developing ML models and deploying them has always been easy. The Syntiant TinyML board was chosen to deploy our model because it provides ultra-low power consumption, a fully connected neural network architecture, an onboard microphone, its tiny size, and is also fully supported by Edge Impulse.

Created By: Attila Tokes

Public Project Link:

Project Demo

Intro

Noise pollution can be a significant problem especially in densely populated urban areas. It can have negative effects both humans and the wildlife. Also, noise pollution is often caused by power hungry activities, such as industrial processes, constructions, flights, etc.

A Noise Pollution Monitoring device built on top of the Nordic Thingy:53 development kit, with smart classification capabilities using Edge Impulse can be a good way to monitor this phenomenon in urban areas. Using a set of Noise Pollution Monitoring the noise / environmental pollution from a city can be monitored. Based on the measured data, actions can be taken to improve the situation. Activities causing noise pollution tend to also have a high energy consumption. Replacing this applications with more efficient solutions can reduce their energy footprint they have.

In this project I will demonstrate how a low power device like the Nordic Thingy:53 can be used in conjunction with an edge machine learning platform like Edge Impulse to build a smart noise / environmental pollution monitor. The PDM microphone of the Nordic Thingy:53 will be used to capture environmental noise. A digital signal processing (DSP) and Inference pipeline built using Edge Impulse will be used to classify audio samples of know activities like construction works, traffic and others.

Getting Started with the Thingy:53

The Nordic Thingy:53 is comes with the pre-installed firmware, that allows us to easily create machine learning projects with Edge Impulse.

To get started with the app we will need to create an Edge Impulse account, and a project:

After this we should be able to detect the Thingy:53 in the Devices tab. The thingy will show up as a device named EdgeImpulse.

Going to the Inference tab we can try out the pre-installed demo app, which uses the accelerometer data to detect 4 types of movement.

Collecting Audio Data

The first step of building a machine learning model is to collect some training data.

For this proof-of-concept, I decided to go with 4 classes of sounds:

• Silence - a silent room

• Nature - sound of birds, rain, etc.

• Construction - sounds from a construction site

• Traffic - urban traffic sounds

A the source of the audio samples I used a number of Youtube videos, listed in the Resources section.

The audio sample can be collected from the Data tab of the nRF Edge Impulse app:

The audio samples are automatically uploaded to the Edge Impulse Studio project, and should show up in the Data Acquisition tab:

By default all the samples will be put in the Train set. We also need a couple of samples for verification, so we will need to run a Train / Test split:

After this we should have approximately 80% of samples in the Train set, and 20% in the Test set:

Training an Audio Classification Model

Having the audio data, we can start building a machine learning model.

In Edge Impulse project the machine learning pipeline is called an Impulse. An impulse includes the pre-processing, feature extraction and inference steps needed to classify, in our case, audio data.

For this project I went will the following design:

• Time Series Data Input - with 1 second windows @ 16kHz

• Audio (MFE) Extractor - this is the recommended feature extractor for non-voice audio

• NN / Keras Classifier - a neural network classifier

• Output with 4 Classes - Silence, Nature, Traffic, Construction

The impulse blocks were trained mostly with the default settings. The feature extraction block looks like follows:

This is followed by the classification block:

The resulting model is surprisingly good:

Most of the test samples were correctly classified. We only have a couple of mismatches for the Traffic / Construction and Silence / Nature classes. This is however expected, as these sounds can be pretty similar.

Deploying the Model on the Thingy:53

Building an deploying an embedded application including machine learning used to involve a couple of steps. With the Thingy:53 and Edge Impulse this become much easier.

We just need to go to the Deployment tab, and hit Deploy. The model will automatically start building:

A couple of minutes later the model is built and deployed on our Thingy:53:

Running Live Inference on the Thingy:53

The Deployment we did earlier should have been uploaded a firmware with the new model on the Thingy:53. Hitting Start on the Inference will start live classification on the device.

I tested the application out with new audio samples for each class:

A Network of Devices

In future versions this project could be extended to also include features like:

• Noise level / decibel measurement

• Cloud connectivity via Bluetooth Mesh / Thread

• Solar panel charging

A network of such monitoring devices could be used to monitor the noise / environmental pollution in a city. Based on the collected data high impact / polluting activities can be identified, and can be replaced with better alternatives.

Resources

Sound Sources:

Created By: David Tischler

Public Project Link:

Introduction

Most industrial settings such as pumping facilities, heating, ventilation, and air conditioning (HVAC) machinery, datacenter infrastructure rooms, manufacturing and heavy industry sites will have proprietary, expensive, critical machinery to maintain. On-site workers with experience near the machinery or equipment, are generally quick to identify when a machine doesn't seem to "sound right", "feel right", or "look right". Previous experience helps them to understand the "normal” state of the equipment, and this intuition and early warning can allow for scheduled repairs or proper planning for downtime. However, facilities that don't have 24/7 staffing or on-site workers could be more prone to equipment failures, as there is no one to observe and take action on warning signs such as irregular sounds, movements, or visual indicators.

Predictive Maintenance using machine learning aims to solve for this problem, by identifying and acting upon anomalies in machinery health.

Hardware

We'll use the onboard microphone of the RaSynBoard to create a machine learning model that understands a "low-speed" sound made by a small pump, and a "high-speed" noise from the pump. For the purposes of this demo, we will consider the "high-speed" noise to be a bad sign, indicating there is a problem.

Reasons for a pump to speed up and attempt to push more liquid could vary, depending on the use-case:

• Liquid cooling, perhaps temperatures are rising and automation has increased the pump flow to try to extract more heat from the system.

• Flow has been restricted, and the pump is trying to achieve the same total volume of throughput but through a smaller diameter.

• Liquid levels are rising, and the pump is trying to reduce the volume of water in a tank, lake, or other holding area.

Dataset Collection

Getting started, we will need to collect audio samples from the pump, running at both low-speed and at high-speed, as well as a variety of “other” background noises that will form an “Unknown” classification, helping us filter out random everyday noises that occur. To collect the data, we’ll first need to prepare the RaSynBoard.

Hardware Setup

With the board connected to the computer, inside the extracted .zip file, run the flash_linux.sh or flash_win.bat file depending upon whether you are on Linux or Windows, and the Edge Impulse firmware will be flashed to the board. When completed, you can disconnect the RaSynBoard from the computer.

Next, insert the SD Card into your laptop or desktop, and copy the config.ini, mcu_fw_120.synpkg, dsp_firmware.synpkg, and ei_model.synpkg` files from the unzipped firmware directory to the SD Card. Upon completion, eject the SD Card from your computer.

Return the Jumper to pins 1-2, and insert the SD Card into the RaSynBoard. Power it back on, and reconnect it to the computer once again.

Connect to Studio

Now that the RaSynBoard is running Edge Impulse firmware, you can use the edge-impulse-daemon command (which got installed as part of the CLI installation) on your desktop or laptop to connect the RaSynBoard to Edge Impulse. The board will then show up in the “Devices” page, and over on the Data Acquisition page you can use the drop-down menus to choose your device, choose the Microphone, enter a sample length of 1000 milliseconds (1 second), and provide a label for the type of data you are collecting. Once ready you will simply click “Start sampling”. (The IMU is also available, for projects that might make us of accelerometer data).

With the pump running in what we’ll call our “normal” condition, low-speed, I’ve collected about 50 audio samples with the RaSynBoard placed on top of the pump motor, each 1 second in length. My label is entered as low-speed, but you can use another term if you prefer. You’ll want to keep the location of the RaSynBoard consistent, so that the data used to train the model is representative of what will be experienced once the model is built and deployed back to the board.

Next, I’ve increased the speed of the pump from 10% to 85%, which still gives some excess pumping capacity, but could be a cause for concern in those use-cases outlined above. The same process is followed, with individual audio samples collected and labeled as high-speed, about 50 samples should be enough.

Building a Model

Once a dataset has been collected, a machine learning model can be built that identifies or recognizes these same sounds in the field. So in this project, we should be able to identify and classify the sound of our pump running at low-speed, and the sound of our pump running at high-speed.

On the left, click on “Impulse design”, and in the first block, “Time Series Data” should already be selected. Add a Processing block, and choose Audio (Syntiant). Then add a Learning block, and choose Classification. Finally, the Output features should already be selected, thus you can click Save Impulse.

Next, on the left navigation, choose Syntiant, to configure the Processing block. This is where Features will be extracted, and you will see a Spectrogram visualization of the processed features for each data sample (chosen from a drop-down menu at the top-right of the raw data). You should be able to leave the default MFE configuration alone, and simply click on Save parameters at the bottom. However, each option has a help tip, should you need to fine-tune these settings. On the next page, click on Generate features to create a Feature explorer that will plot a graph, allowing you to examine how your Features cluster.

Choose Classifier on the left navigation to proceed to the neural network settings for the Learning block, where you can choose the number of training cycles (epochs), learning rate, and other settings (again the defaults will likely work fine, but help text can guide any fine-tuning you may want to perform). Once you are ready, you can click on Start training to begin the model build process.

Upon completion, you will see another visual representation, this time evaluating the validation performance of your model and the Accuracy against a validation set.

Deploying a Model

To deploy this newly-built audio classification machine learning model onto the RaSynBoard, there are a few options. The quickest way to test if your model is functional and determine accuracy in the real world, is to simply choose a ready-made firmware binary that can be flashed and run immediately on the RaSynBoard. However, when building a full-featured application intended for production use, you’ll instead want to choose either the Syntiant NDP120 library or Custom Block export options, as you can then leverage the machine learning model in your application code as necessary, depending on your use-case or product.

For now though, I will just use the binary download as an easy way to test out the model.

On the left navigation, click on Deployment, and type “RaSynBoard” in the search box. From the options, choose RaSynBoard and then click on the Find posterior parameters button, which will enable you to choose the keywords or sounds from your labels that you want to detect. Choose “low-speed” and “high-speed”, then click on Find parameters.

Next, click on Build, and the firmware will be generated and downloaded to your computer. Once downloaded, unzip the file, and we’ll follow a similar method as earlier.

Power down the RaSynBoard if it is still running, remove the SD Card from the board and insert the SD Card into your laptop or desktop, and copy the config.ini, mcu_fw_120.synpkg, dsp_firmware.synpkg, and ei_model.synpkg files from the unzipped download to the SD Card. Upon completion, eject the SD Card from your computer and return it to the RaSynBoard.

Inference Results

We can now power the RaSynBoard back on, the board will boot up and automatically start running the model. To see the results, we need to attach to a serial console and can view the output of any inference results. Using a standard UART adapter, connect Ground, TX, and RX to pins 2,4, and 6 on the I/O Board, as shown here:

Then in a terminal, you will see the output of the model running. I have placed the RaSynBoard back on the pump, set the speed to low, and sure enough, the model is able to predict the pump is running at low-speed. Increasing the compressor power to 85%, the RaSynBoard now recognizes that the pump is running at high-speed.

Going Further

Using machine learning and an embedded development kit, we were able to successfully identify and classify pump behavior by listening to the sound of the compressor. This demonstration validated the approach as feasible, and when wrapped into a larger application and alerting system, an audio classification model could be used for remote infrastructure facilities, factory equipment, or building HVAC equipment that is not continually monitored by workers or other human presence. The Renesas RA6 MCU combined with the Syntaint NDP120 neural decision processor in the Avnet RaSynBoard create a low-power, cost-effective solution for predictive maintenance or intervention as needed, prior to a failure or accident occurring.

Created By: Samuel Alexander

Public Project Link:

GitHub Project Link:

Introduction

In the era of smart devices and voice-controlled technology, developing effective keyword spotting (KWS) systems is crucial for enhancing user experience and interaction. This documentation provides a comprehensive guide for creating a portable LED product with KWS capabilities, using Edge Impulse's end-to-end synthetic data pipeline. Synthetic voice/speech data, generated artificially rather than collected from real-world recordings, offers a scalable and cost-effective solution for training machine learning models. By leveraging AI text-to-speech technologies, we can create diverse and high-quality datasets tailored specifically for our KWS applications. This guide not only serves as a blueprint for building a responsive LED product but also lays the groundwork for a wide range of voice-activated devices, such as cameras that start recording on command, alarms that snooze with a keyword, or garage doors that respond to voice prompts.

Problem Exploration

Traditional methods of training keyword spotting models often rely on extensive datasets of human speech, which can be time-consuming and expensive to collect. Moreover, ensuring diversity and representation in these datasets can be challenging, leading to models that may not perform well across different accents, languages, and speaking environments. Synthetic data addresses these challenges by providing a controlled and flexible means of generating speech data. Using AI text-to-speech technology, we can produce vast amounts of speech data with varied voices, tones, and inflections, all tailored to the specific keywords we want our models to detect.

This approach opens up numerous possibilities for product development. For instance, a smart LED light can be designed to turn on or off in response to specific voice commands, enhancing convenience and accessibility. A camera can be programmed to start recording or take a group photo when a designated keyword is spoken, making it easier to capture moments without physical interaction. Similarly, an alarm system can be configured to snooze with a simple voice command, streamlining the user experience. By utilizing synthetic data, developers can create robust and versatile KWS models that power these innovative applications, ultimately leading to more intuitive and responsive smart devices.

Project Overview

This project outlines the creation of a keyword spotting (KWS) product using Edge Impulse's synthetic data pipeline. The process involves generating synthetic voice data with OpenAI's Whisper text-to-speech model via Edge Impulse Studio and training the KWS model using Syntiant's audio processing blocks for the NDP120 on the Arduino Nicla Voice. The phrase 'illuminate' and 'extinguish' will be generated and used for training the model.

After training, the model is deployed onto the Arduino Nicla Voice hardware. A custom PCB and casing are designed to incorporate LED lights and power circuitry, ensuring portability and ease of use. This guide serves as a practical resource for developers looking to implement KWS functionality in voice-activated devices, demonstrating the efficiency of synthetic data in creating responsive and versatile products.

Hardware selection: Arduino Nicla Voice

The Arduino Nicla Voice is an ideal choice for this project due to its use of the Syntiant NDP120, which offers great power efficiency for always-on listening. This efficiency allows the NDP120 to continuously monitor for keywords while consuming minimal power, making it perfect for battery-powered applications. Upon detecting a keyword, the NDP120 can notify the secondary microcontroller, Nordic Semiconductor nRF52832, which can then be programmed to control the lighting system. The compact size of the Nicla Voice also makes it easy to integrate into a small case with a battery. Furthermore, the Nicla Voice's standardized footprint simplifies the prototyping process, allowing for the easy creation of a custom PCB module with LED circuitry that can be easily connected.

Hardware Requirements

• Arduino Nicla Voice (or other Edge Impulse supported MCU with mic)

• PCB and SMD components (parts breakdown explained later)

Software Requirements

• Edge Impulse CLI

• Arduino IDE

• OpenAI API account

Dataset Collection

Creating an OpenAI API Secret Key

Once you have your secret key, you can navigate to your Edge Impulse organization page and enter your API key. Please also note that this Text to Speech Data Generation feature is only available for Edge Impulse Enterprise accounts.

Generating TTS Synthetic Data

Now that we have the environment configured, and our OpenAI API saved in the Edge Impulse Studio, we are ready to start a new project and begin generating some synthetic voice data.

On your project's page select Data acquisition --> Data sources --> + Add new data source --> Transformation block --> Whisper Synthetic Voice Generator --> Fill out the details as follow:

Phrase: illuminate

We need to generate speech for the words "illuminate" and "extinguish", we can start with illuminate for this 'action' first and then set up another data source 'action' for extinguish after we finish configuring this one.

Label: illuminate

Match the label name to the voice sample we want to generate.

Number of samples: 10

We can start with generating 10 samples to test creating this action, if everything works feel free to generate an action that generates more than 10 samples. Once created you can also run the action multiple times.

Voice: random

We want to create diversity of voice and accent in our dataset, so choose random.

Model: tts-1

I tested tts-1 and tts-1-hd, I think the quality difference is negligible for this case, but feel free to select either one. Note that tts-1-hd will cost you more OpenAI credits.

Speed: 0.6, 0.7, 0.8, 0.9, 1.0, 1.1, 1.2

We want to vary the speed of the voice pronouncing the word we want to generate. 0.6 means 60% of its original speed, and 0.6 - 1.2 should give enough range.

Now you can run the action. If successful, the tts voice generation should begin and it may take a few minutes to complete. If the job failed you should be notified and you can recheck if the API key is entered correctly, then you can retry again.

Once satisfied with all the data generated, perform a Train / Test split into approximately 80/20 ratio.

Impulse Design

The Impulse design values are chosen for optimal keyword spotting performance. The 968 ms window size captures enough audio for accurate detection, while the 500 ms window increase balances responsiveness and efficiency. The 16000 Hz frequency is standard for capturing human voice, ensuring quality without excessive data. Using the Audio (Syntiant) block leverages the NDP120's capabilities for efficient digital signal processing. The Classification block distinguishes between commands, with output classes "extinguish," "illuminate," and "z_openset" allowing for control of the lighting system and handling unknown inputs.

• Window size: 968 ms

• Window increase: 500 ms

• Frequency: 16000 Hz

• Audio (Syntiant)

• Classification

• Output: extinguish, illuminate, z_openset

Under Classifier, set the learning rate to 0.0005 and change the architecture preset to use Dense Network.

Our audio classifier gives in accuracy of 93.8% which is satisfactory. We can continue tuning the hyperparameters and try using some data augmentation, but for the purpose of this demonstration we are satisfied with the current result and can move to the deployment phase.

Deployment

Now the AI model is ready to be deployed to the Arduino Nicla Voice. Let's select the Arduino Nicla Voice deployment.

Please note that we want to flash the firmware that you have just built, instead of the default audio firmware for the Nicla Voice's NDP120.

Once the code is uploaded you can verify if everything works by saying out loud the words 'Illuminate' and 'Extinguish'. When the keyword 'Illuminate' is detected, the blue built-in led will blink and when the keyword 'Extinguish' is detected, the red built-in led will blink.

Next we will design and manufacture the PCB socket which holds the LED and power circuitry which can turn on when 'Illuminate' is detected and turn off when 'Extinguish' is detected.

Designing and Building the KWS Product

The schematic and PCB is designed using KiCAD. A single sided aluminum PCB is selected for this project due to its excellent thermal conductivity, which helps dissipate heat generated by the LEDs and other components, ensuring reliable performance and longevity. The design of this PCB is simple enough to make it possible to route using one side only.

The schematic, pcb, and gerber (manufacturing) files are accessible in the project's github page.

Conclusion

Synthetic data has demonstrated its value in the development of voice-activated products like Lumo Voice. Its customizable nature allows for the inclusion of diverse accents, languages, tones, and inflections, resulting in robust keyword spotting models that perform well across different speaking styles. Unlike traditional data collection, which requires gathering numerous samples from various individuals, synthetic data offers a more efficient and scalable approach, enabling the rapid generation of high-quality datasets tailored to specific needs. This flexibility, combined with Edge Impulse's incredible streamlined workflow, has made it easier than ever to build and deploy small, efficient models on edge devices. With Edge Impulse, we could quickly generate synthetic data, train, and optimize our models, making it a powerful tool for creating responsive and versatile voice-controlled devices like Lumo Voice.

Created By: Pratyush Mallick

Public Project:

Intro

This project focuses on how to port an existing audio recognition project built with a SiLabs Thunderboard Sense 2, to the latest as used in the newer SiLabs xG24 Dev Kit. For demonstration purposes, we will be porting project, which is an Edge Impulse based TinyML model to predict various vehicle failures like faulty drive shaft and brake-pad noises. Check out his work for more information.

The audio sensor on the Thunderboard Sense 2 and the xG24 Dev Kit are the same (TDK InvenSense ICS-43434), so ideally we're not required to collect any new data using the xG24 Dev Kit for the model to work properly. Had the audio sensor been a different model, it would most likely be necessary to capture a new dataset.

However, note has to be taken that the xG24 has two microphones, phones placed at the edges of the board.

In this project, I am going to walk you through how you can clone Mani's Public Edge Impulse project for the Thunderboard Sense 2 board, build it for the xG24, test it out, and then deploy to the newer SiLabs xG24 device instead.

Installing Dependencies

Before you proceed further, there are few software packages you need to install.

Clone And Build

Click on the "Clone" button at top-right corner of the page.

That will bring you to the below popup tab. Enter a name for your clone project, and click on the "Clone project" button.

This action will copy all the collected data, generated features, and model parameters into your own Edge Impulse Studio. You can verify this by looking at the project name you entered earlier.

Now if you navigate to "Create impulse" from the left menu, you will see how the model was created originally.

As you can see, the model was created based on audio data sampled at 16KhZ. As mentioned, because the audio microphones used on the both board boards are the same, we're not required to collect any additional data from the new board.

However, if you do want to collect some data from the xG24, then you will need to flash the base firmware and then use the edge-impulse-daemon to connect the device to the Studio.

You can follow the guide below to go through the process, if you are interested in adding more data samples to your cloned project:

With default value of Window Size (10s) and Window Increase (500 ms), the processing block will throw an error, as represented below:

This is because some of the features in Edge Impulse's processing block have been updated since this project was created, so you need to update some of the parameters in the Timer Series block such as Window Size and Window Increase, or increase the frame stride parameter in the MFE processing block. This is what my updated window parameters look like:

Next, navigate to the "Classification" tab from the left menu, and click on "Start training".

Alternatively, you can also collect more data as mentioned above, or add new recognized sounds with other audio classes, then begin your training.

Test

When you are done training, navigate to the "Live Classification" page from the left menu. This feature of Edge Impulse comes in handy when migrating projects to different boards.

Rather than deploying the model and then testing it on the hardware, with this feature we can actually collect audio data from the hardware immediately, and run the model in the Studio on the collected data. This saves time and effort before hand.

For Edge Impulse supported boards we can directly download the base Edge Impulse firmware, and then directly record audio (or other) data from the target device.

Once done, you can select the device name, select the sensor as "Microphone", sample length and the sampling frequency (ideally equally to collected samples).

Deploy

When you are done retraining, navigate to the "Deployment" tab from the left menu, select "SiLabs xG24 Dev Kit" under "Build firmware", then click on the "Build" button at the bottom of the page.

This will build your model and download a .zip file containing a .hex file and instructions.

With the Thunderboard Sense 2 deploying firmware could be done by directly dragging and dropping files to the "USB Driver TB004" when the device was connected in flash mode to a host PC. However, for the xG24 we have to use Simplicity Commander to upload the firmware to the board. You need to first connect the xG24 board the PC, make note of the COM port for the board, and ideally it will be identified by the PC as a J-Link UART port.

Now open the Simplicity Commander tool and connect the board. Once connected, select the "Flash" option on the left and then select the downloaded .hex file and flash it to the board.

To start the inferencing run the following command in your terminal:

Note that this is a newer command supported by the Edge Impulse CLI, hence you may need to update your edge-impulse-cli version to get this running and avoid a package mismatch as shown below:

Now your model should be running, and recognize the same audio data and perform inferencing on the newer xG24 Dev Kit hardware, with little to no modifications to actual data or to the model architecture.

This highlights the platform agnostic nature of Edge Impulse, and was possible in this case because the audio sensor on both the Thunderboard and xG24 are the same. However, you would need do your own due diligence for migrating projects built with other sensor data such as humidity/temperature, or the light sensor, as those do vary between the boards.

One final note is that in this project, the xG24 is roughly 2x as fast as the Thunderboard Sense 2 in running the DSP, and 8x faster in running the inference:

Hopefully this makes upgrading your SiLabs projects easier!

Created By: Jallson Suryo

Public Project Link:

GitHub Repo:

Introduction

Can you imagine Amazon Alexa without the cloud?

For most services, adding voice means adding an internet connection and it means extra expense, privacy and security concerns, and you need to install an app (or access to the web) for everything in your home. Another problem is the time delay between a voice command being given, the command being sent to a cloud server and then back to the device for execution, creating a poor user experience.

Our Solution

A power-plug which can be controlled using voice commands, with no connection to the internet. By using a machine learning model embedded in a microcontroller that has been trained to recognize several commands, which are then passed to relays which will turn the power on or off at each socket according to the issued command, instantly. Practicality, privacy concerns, and cost-effectiveness are the goals of this Non-IoT Voice Controlled Power Plug project.

Description

This project takes advantage of Edge Impulse's Syntiant audio processing block that extracts time and frequency features from a signal, specific to the Syntiant NDP120 accelerator included in the Nicla Voice. The NDP120 is ideal for always-on, low-power speech recognition applications with the “find posterior parameters” feature that will only react to the specified keywords.

Devices with an embedded ML model will accept voice commands, but won't need a WIFI or Bluetooth connection. All processing is done locally on the device, so you can directly tell a lamp, air conitioner, or TV to turn on or off without Alexa or Siri, or any digital assistant speaker/hub.

This project will use relays and a power strip connected to various appliances such as a lamp, fan, TV, etc. An Arduino Nicla Voice with embedded ML model has been trained to recognize various keywords like: one, two, three, four, on, and off is the center of the decision process. From the Nicla Voice we use the I2C protocol which is connected to an Arduino Pro Micro to carry out voice commands from the Nicla Voice, and forwarded to the relays which control power sockets.

Hardware Requirements:

• Arduino Nicla Voice (with Syntiant NDP120)

• Any microcontroller or Arduino (I use Pro Micro)

• 5V Relay (4pcs)

• Breadboard

• Cable jumper

• Cable for 110/220V

• Powerstrip (4 sockets)

Software & Online Services:

• Edge Impulse Studio (Enterprise account or Free Trial for more than 4 hours of training data)

• Arduino IDE

• Terminal

Steps

1. Upload or Collect Audio Data

Before we start, we need to install the Arduino CLI and Edge Impulse tooling on our computer.

2. Split and Balancing

Click on a data sample that was collected, then click on the 3 dots to open the menu, and finally choose Split sample. Set the segment length to 1000 ms (1 second), or add segments manually, then click Split. Repeat this process until all samples are labeled in 1 second intervals. Make sure the comparison between one, two, three, four, on, off and unknown data is quite balanced, and the ratio between Training and Test data is around 80/20.

3. Train and Build the Model (Syntiant)

Choose Create Impulse, set Window size to 968ms, then add an Audio (Syntiant) Processing block, and choose Classifier for the Learning block, then Save Impulse. In the Syntiant parameters, choose log-bin (NDP120/200) then click Save. Set the training to around 50 cycles with 0.0005 Learning rate, and choose Dense Network with Dropout rate around 0.5, then click Start training. It will take a short while, but you can see the progress of the training on the right. If the results show a figure of around 80% accuracy upon completion, then we can most likely proceed to the next stage.

Now we can test the model in Live classification, or choose Model testing to test with the data that was set aside earlier (the 80/20 split), and click Classify all. If the result is quite good -- again around 80% accuracy, then we can move to the next step -- Deployment.

4. Deploy to Nicla Voice

5. Upload the Arduino Code

Because there are two MCU's in this solution, two seperate applications are needed:

Nicla Voice

Pro Micro

Demo

Conclusion

Finally we succeeded in making this Non IoT Voice Controlled Power-plug idea a reality and implemented in a home appliances setting. I believe in the future this kind of non-IoT smart home system will be widely implemented, and could be built-in to every home appliance with specific keywords. Concerns about privacy, security, as well as practicality and energy efficiency can be achieved for a more sustainable future.

Created By: Solomon Githu

Public Project Link:

GitHub Repository:

Introduction

It's said that a dog is man's best friend and it is no secret that dogs are incredibly loyal animals. They are very effective when it comes to securing premises, and are also able to sense when things are not right, whether with a person or with a situation. Some examples of dog security are guidance for people with visual impairments, detection of explosives and drugs, search and rescue missions, and enhancing security at various places. Worker safety aims to foster a practice of ensuring a safe working environment by providing safe equipment and implementing safety guidelines that enable workers to be productive and efficient in their job. In this case, dogs are usually deployed to patrol and monitor areas around workplace. One of the reasons is because dogs have an extraordinary sensing ability of smell, vision and hearing; making them exceptional at detecting threats that may go unnoticed by humans or other security systems. However, workers may not always be able to interpret a dog's barks in time. The workers may not be knowledgeable of how dog's react, or they may be focusing on their tasks and fail to hear a dog. Failure to detect a dog's bark may lead to fatalities, injuries or even accidents.

Machine listening refers to the ability of computers to understand audio signals similarly to how humans hear and understand various sounds. Recently, labeling of acoustic events has emerged as an active topic covering a wide range of applications. This is because by analyzing animal sounds, AI can identify species more accurately and efficiently than ever before and provide unique insights into the behaviors and habitats of animals without disturbing them. Barking and other dog vocalizations have acoustic properties related to their emotions, physiological reactions, attitudes, or some other internal states. Historically, humans have relied on intuition and experience to interpret these signals from dogs. We have learned that a low growl often precedes aggression, while a high-pitched bark might indicate excitement or distress. Through this experience, we can train AI models to recognize dog sounds, and those who work with the animals— like security guards, maintenance staff, and even delivery people can use that insight.

The AI model only requires to be trained to recognize the sounds one seeks to monitor based on recordings of the sound. However, creating an audio dataset of animal sounds is quite challenging. In this case, we do not disturb dogs, or other animals, to provoke reactions like barking. Fortunately, Generative AI is currently at the forefront of AI technology. Over the past decade, we have witnessed significant advancements in synthetic audio generation. From sounds to songs, with just a simple prompt, we can now use computers to generate dog sounds and in turn use the data to train another AI model.

Project Overview

This project aims to develop a smart prototype wearable that can be used by workers to receive alerts from security dogs. In workplaces and even residential areas, dog sounds are common, but we often overlook them, assuming there is no real threat. We hear the sounds but don't actively listen to the warnings dogs may be giving. Additionally, workers at a site may be too far to hear the dogs, and in some cases, protective ear muffs further block out environmental sounds.

Sound classification is one of the most widely used applications of Machine Learning. This project involves developing a smart wearable device that is able to detect dogs sounds specifically barking and howling. When these dogs sounds are detected, the wearable communicates about the dog's state by displaying a message on a screen. This wearable can be useful to workers by alerting them of precautionary measures. A security worker may be alerted of a potential threat that a dog identified but they did not manage to see. A postal delivery person can also be alerted of an aggressive dog that may be intending to attack them as they may perceive the delivery person as a threat.

To train a Machine Learning model for this task, the project uses generative AI for synthetic data creation. The reason why I chose this is because we cannot distress a dog so that we can obtain reactions like barking or howling. I also wanted to explore how generative AI can be used for synthetic data generation. Ideally, when training Machine Learning models, we want the data to be a good representation of how it would also look when the model is deployed (inference).

Use Case Explanation

Canine security refers to the use of trained security dogs and expert dog handlers to detect and protect against threats. The effectiveness in dogs lies in their unique abilities. Animals, especially dogs, have a keen sense of smell and excellent hearing. As a result, dogs are the ideal animal to assist security guards in their duties and also provide security to workplaces and homesteads. However, at the same time, according to the American Veterinary Medical Association, more than 4.5 million people are bitten by dogs each year in the US. And while anyone can suffer a dog bite, delivery people are especially vulnerable. Statistics released by the US Postal Service show that 5,800 of its employees were attacked by dogs in the U.S. in 2020.

According to Sam Basso, a professional dog trainer, clients frequently admit they have more to learn about their dogs during his sessions. While humans have been able to understand how dogs act, there is still more learning that the average person requires so that they can better understand dogs. There are professional dog handlers that can be used to train owners but this comes at a great cost and also not everyone is ready to take the classes. To address these issues, we can utilize AI to develop a device that can detect specific dog sounds such as barking, and alert workers so that they can follow up on the situation that the dog is experiencing. In the case of delivery persons, an alert can inform them of a nearby aggressive dog.

Audio classification is a fascinating field with numerous applications, from speech recognition to sound event detection. Training AI models has become easier by using pre-trained networks. The transfer learning approach uses a pretrained model which is already trained using a large amount of data. This approach can significantly reduce the amount of labeled data required for training, it also reduces the training time and resources, and improves the efficiency of the learning process, especially in cases where there is limited data available.

Finally, for deployment, this project requires a low-cost, small and powerful device that can run optimized Machine Learning models. The wearable will also require ability to connect to an OLED display using general-purpose input/output (GPIO) pins. Power management is another most important consideration for a wearable. The ability to easily connect a small battery, acheive low power consumption, and have battery charging would be great. In this case, the deployment mode makes use of the XIAO ESP32S3 development board owing to it's small form factor, high performance and lithium battery charge management capability.

Components and Hardware Configuration

Software components:

• Edge Impulse Studio account

• ElevenLabs account

• Arduino IDE

Hardware components:

• A personal computer

• SSD1306 OLED display

• 3.7V lithium battery. In my case, I used a 500mAh battery.

• Some jumper wires and male header pins

• Soldering iron and soldering wire

• Super glue. Always be careful when handling glues!

Data Collection Process

Once we have an account on both ElevenLabs and EdgeImpulse, we can get started with data creation. First, create a project (with Professional or Enterprise account) on Edge Impulse Studio. On the dashboard, navigate to "Data acquisition" and then "Synthetic data". Here, we will need to fill the form with our ElevenLabs API key and also parameters for the data generation such as the prompt, label, number of samples to be generated, length of the each sample, frequency of the generated audio files, and also prompt influence parameter.

To get an API key from ElevenLabs, first login to your account. Next, on the "home" page that opens after logging in, click "My Account" followed by "API Keys". This will open a new panel that enables managing the account API Keys. We need to click "Create API Key" and then give a name to the key, although the naming does not matter. Next, we click the "Create" button and this will generate an API key (a string of characters) that we need to copy to our Edge Impulse project in the "ElevenLabs.io API Key" field.

For this demonstration project, at first I worked with 3 classes of sounds: dog barking, dog howling and environment sounds (with city streets, construction sites and people talking). In this case, the prompt that I used to generate sound for each class was "dog howling", "dog barking" and "environmental sounds (e.g., city streets, construction sites, people talking)" respectively. The labels for each class was dog_howling, dog_barking and environment respectively. For each prompt, I used a prompt influence of 0.6 (this generated the best sounds), "Number of samples" as 6, "Minimum length (seconds)" as 1, "Frequency (Hz)" as 16000 and "Upload to category" as training. With these configurations, when we click the "Generate data" button on Edge Impulse Studio, this will generate 6 audio samples each of 1 second for one class. To generate sound for another class, we can simply put the prompt for it and leave the other fields unchanged. I used this configuration to generate around 39 minutes of audio files consisting of dogs barking, dogs howling and environment (e.g., city streets, construction sites, people talking) sounds.

In Machine Learning, it is always a challenge to train models effectively. Bias can be introduced by various factors and it can be very difficult to even identify that this problem exists. In our case, since the device will be continuously recording environment sound and classifying it, we need to also consider that it's not always that there will be a dog barking, dog howling, people talking and city sounds present. The environment can also be calm, with low noise or other sounds that the generative AI model failed to include in the environment class. Identifying this is key to fixing the bias of using the environment sounds (e.g., city streets, construction sites, people talking).

Finally, after using ElevenLabs integration and uploading my noise sound recording, I had around 36 minutes of sound data for both training and testing. In AI, the more data, the better the model will perform. For this demonstration project, I found the dataset size to be adequate.

We then click the button "Perform train / test split" on the interface that opens. This will open another interface that asks us if we are sure of rebalancing our dataset. We need to click the button "Yes perform train / test split" and finally enter "perform split" in the next window as prompted, followed by clicking the button "Perform train / test split".

Training the Machine Learning Model, with Experiments

By using the Autotune feature on Edge Impulse, the platform updated the processing block to use frame length of 0.025, frame stride of 0.01, filter number of 41, set the Lowest band edge of Mel filters (in Hz) to 80 and set the Noise floor (dB) as -91.

The last step is to train our model. We click "Classifier" which is our learning block that is using a Convolution Neural Network (CNN) model. After various experiments, I settled with 100 training cycles, a learning rate of 0.0005 and a 2D Convolution architecture. Finally, we can click the "Save & train" button to train our first model. After the training process was complete, the model had an accuracy of 98% and a loss of 0.04.

When training our model, we used 80% of the data in our dataset. The remaining 20% is used to test the accuracy of the model in classifying unseen data. We need to verify that our model has not overfit, by testing it on new data. To test our model, we first click "Model testing" then "Classify all". Our current model has an accuracy of 99%, which is pretty good!

At last, we have a simple Machine Learning model that can detect dog sounds! However, how do we know if this configuration is the most effective? We can experiment with three other processing blocks: Spectrogram, raw data processing and MFCC. To be specific, the difference between the Impulses in this project is the processing block. To add another Impulse, we click the current Impulse (Impulse #1) followed by "Create new impulse".

Deploying the Impulses to XIAO ESP32S3

In this project, we now have four Impulses. The Experiments feature not only allows us to setup different Machine Learning processes, but it also allows us to deploy any Impulse. The MFE, Spectrogram and MFCC Impulses seem to perform well according to the model training and testing. I decided to skip deploying the Raw data Impulse since using raw data as the model input does not seem to yield good performance in this use case.

We can then deploy the second Impulse which uses the Spectrogram pre-processing algorithm. The steps for the deployment are similar - we select Arduino library, enable EON compiler, select Quantized (int8) and download the Arduino library. To speed up compilation and use cached files in the Arduino IDE, we can simply unzip the second Impulse Arduino library and copy over the model-parameters and tflite-model folders to the first Impulse's Arduino library folder, overwriting the existing files with the updated model parameters. Unfortunately, the model is not able to run on the ESP32S3 board and we get an error failed to allocate tensor arena. This error means that we have run out of RAM on the ESP32S3.

Lastly, I experimented with deploying the MFCC Impulse. This algorithm works best for speech recognition but the model training and testing show that it performs well for detecting dog sounds. Following similar steps, I deployed the fourth Impulse using the EON Compiler and Quantized (int8) model optimizations. Surprisingly, this Impulse (using the MFCC processing algorithm) delivers the best performance even compared to the MFE pre-processing block. The Digital Signal Processing (DSP) takes approximately 285ms, with classification taking about 15ms. For detecting dog sounds, this Impulse accurately identifies with great confidence, demonstrating the positive impact of a DSP block on model performance!

Based on the experiments, I chose to continue with the fourth Impulse due to its accuracy and reduced latency.

Assembling the Wearable

A solid gadget needs a solid case! We are close, so its time to put our wearable together.

The other 3D printed components are two flexible wrist straps. These are similar to the ones found on watches. I achieved the flexibility by printing them with TPU material. Note that if you do not have a good 3D printer you may need to widen the strap's holes after printing. I used super glue to attach the wrist straps to the case. Always be careful when handling glues!

A cool part is the wearable's dock/stand. This component is not important to the wearable's functionality, but a device's dock/stand is just always cool! It keeps your device in place, adds style to your space, and saves you from the fear of your device being tangled in cables.

The wearable's electronic components include:

• Seeed Studio XIAO ESP32S3 (Sense) development board with the camera detached

• SSD1306 OLED display

• 3.7V lithium battery. In my case, I used a 500mAh battery.

• Some jumper wires and male header pins

The XIAO ESP32S3 board has a LiPo battery connector copper pads that we can use to solder wires for the battery connection. Note that the negative terminal of the power supply is the copper pad closest to the USB port, and the positive terminal of the power supply is the copper pad further away from the USB port.

Once the electronic parts have been assembled, they can be put in the wearable's case according to the layout in the image below. Side vents on the case allow the onboard digital microphone to capture surrounding sounds effectively and they also help cool the ESP32S3.

Below is an image of my wearable after assembling the components.

Result

To test the wearable, I used a television to play YouTube videos of construction sites and dog sounds. At first, I was expecting the model to not perform well since the YouTube playback sounds were not the same as the sounds that were generated by ElevenLabs. In Machine Learning, we target to train the model on a dataset that is the same representation of what it will be seeing during deployment. However, the model, and using the MFCC algorithm, performed well and it was able to accurately detect dog sounds, though sometimes barks were classified as howls and vice versa.

Let’s now put on our safety hats, or get packages to deliver, and put the TinyML wearable to test.

Conclusion

This low cost and low-power environmental sensing wearable is one of the many solutions that embedded AI has to offer. The presence of security dogs provides a sense of security and an unmatched source of environment state feedback to us humans. However, there is a great need to also understand how these intelligent animals operate so that we can understand and treat them better. The task at hand was very complicated, to capture sounds without causing disturbance to dogs, train a dog sound detection model, and optimize the model to run on a microcontroller. However, by utilizing the upcoming technologies of synthetic data generation and powerful tools offered by the Edge Impulse platform; we have managed to train and deploy a custom Machine Learning model that can help workers.