1 of 16

Air Quality and Environmental Projects

Arduino x K-Way - Environmental Asthma Risk Assessment

Researching the relationship between temperature, humidity, and asthma with wearable tech on the Arduino Nicla x K-Way smart jacket.

Created By: Nick Bild

Public Project Link: https://studio.edgeimpulse.com/public/148301/latest

GitHub Repository: https://github.com/nickbild/environmental_asthma_risk

Intro

The US economy takes a hit of more than 80 billion dollars annually due to both direct and indirect effects of asthma. This comes in the form of medical expenses, missed work days, and even death. Some of these impacts can be prevented by avoiding asthma triggers and taking preventative measures, like using prescribed inhalers or limiting physical activity when at risk.

One thing that may help in understanding when someone is at high risk for an asthma attack is monitoring environmental risk factors. It has been noted, for example, that factors like temperature and humidity increase the number of emergency department visits for the condition. But the exact relationships between weather conditions and asthma flare ups are not entirely clear.

I decided to build a machine learning model and train it to understand the relationship between temperature, humidity and emergency department visits for asthma. Since a device running this model would need to always be with the person being monitored, making it into a wearable gadget makes a lot of sense. As it turns out, K-Way and Arduino recently teamed up to make a smart jacket that looks like the perfect platform to build my idea on, so I gave it a try.

Note that this device has not been validated clinically, nor has it been approved by the FDA or any other regulatory agency. It is a proof of concept and cannot be used to make health-related decisions.

Hardware Requirements

1x K-Way jacket with integrated Arduino Nicla Sense ME

Software Requirements

Edge Impulse Studio
Arduino IDE

How It Works

The K-Way jacket is instrumented with an Arduino Nicla Sense ME. The Arduino is housed inside a tiny case and comes wired to a rechargeable LiPo battery. This hardware platform was designed with tinyML in mind, with an Arm Cortex M4 CPU operating at 64 MHz to run local inferences, and a slew of sensors, including a number of motion and environmental sensors.

One of the sensors on this board collects temperature and humidity measurements, then passes them into a machine learning model that I built with Edge Impulse Studio that predicts the number of emergency department visits for asthma that would be expected under those conditions. If that number exceeds a certain threshold, that is considered a high-risk day for asthma flare ups, and a message is sent to the jacket wearer’s smartphone to give them a heads up so they can take appropriate action.

Data Preparation

I located a dataset that provides hourly weather and asthma emergency department visit metrics for an entire year. I processed this data with a simple Python script to create CSV files compatible with Edge Impulse. These CSV files were uploaded to my project in Edge Impulse Studio using the data acquisition tool.

Building the ML Model

I designed a simple impulse that feeds the previously uploaded training data into a regression model that learns to translate temperature and humidity into a prediction of the number of emergency department visits expected for the hour under those conditions.

Model testing showed that the model was accurate over 73% of the time.

Deploying the Model

Edge Impulse offers many options for deployment, but in my case the best option was the "Arduino library" download. This packaged up the entire classification pipeline as a compressed archive that I could import into Arduino IDE, then modify as needed to add my own logic. That allowed me to send Bluetooth Low Energy messages when certain thresholds were met. The Arduino sketch is available here.

Conclusion

The K-Way/Arduino smart jacket is an interesting new platform for tinyML applications. By embedding the hardware in a jacket, it does not require any special effort on the part of the wearer to bring their intelligent algorithms along with them wherever they go. This type of always-available platform is very promising for health-related applications, and worked quite well for my prototype. I hope to see devices like this be clinically validated in the future so that they can make a real difference in people’s day-to-day lives.

Gas Detection in the Oil and Gas Industry - Nordic Thingy:91

Using a Nordic Thingy:91 to detect the presence of gases such as CO2, CO, and Isopropanol.

Created By: Zalmotek

Public Project Link:

https://studio.edgeimpulse.com/studio/127759

GitHub Repository:

https://github.com/Zalmotek/edge-impulse-gas-detection-thingy-91-nordic

Introduction

Gas detection is critical for ensuring the safety of workers in the oil and gas industry. Gas leaks can occur at any stage of production, from drilling and refining to transportation and storage. Gas sensors must be able to detect a wide range of gasses, including combustible gasses like methane and propane, as well as toxic gasses like carbon monoxide and hydrogen sulfide.

Gas detection systems have traditionally been based on point sensors, which are placed at strategic locations throughout a facility. These sensors are connected to a central control panel, which monitors the gas concentration in each location. If a gas leak is detected, an alarm is sounded and workers are evacuated from the area.

The Challenge

Even if they are the industry standard, traditional gas detection systems based on point sensors have several limitations.

They are limited by their space density, as they can only detect gas in the areas where they are positioned, meaning that leaks in other parts of the facility may go undetected.
They are limited by network coverage. Many wireless communication-dependent sensors are prone to malfunction during severe weather events, leaving unsupervised locations in the facility.
These systems rely on human operators to evaluate the data and take action.

Our Solution

To overcome those challenges, we propose a solution based on the Nordic Semi Thingy:91, an IoT multi-sensor device, equipped with an environmental sensor suite, LTE connectivity, and full compatibility with Edge Impulse machine learning models.

Thingy:91 is a complete prototype development platform for cellular IoT applications. It’s an easy-to-use prototyping tool that lets you quickly and cheaply try out your ideas and iterate until you have a working product. The onboard environmental sensor suite includes temperature, humidity, atmospheric pressure, color, light intensity, and UV index sensors. The onboard air quality sensor can detect a wide range of gasses, including methane, carbon monoxide, and hydrogen sulfide.

Gas sensor data is often noisy and unreliable, making it difficult to interpret. Machine learning algorithms can be used to filter out false readings and identify patterns in the data that indicate the presence of certain gasses.

By using the Thingy:91 as an edge device, it can measure the gas concentration, run a machine learning algorithm and decide if there is a trend in the quantity of dangerous gasses in the air and act out without human intervention either by sounding an alarm, either by sending a message to the nearby employees that might be in danger.

A monitoring system based on the Nordic:91 could be used in two manners, each with its own advantages and disadvantages:

As a static point sensor.

This is advantageous when the area in which the sensor is placed is inaccessible via cable connection and is prone to connectivity problems. Gas detection is often performed in remote or difficult-to-access locations, making it impractical to send data to the cloud for analysis. Edge computing allows data to be processed locally, in real time, by using machine learning algorithms.

As a wearable.

By using it as a wearable, it will be able to detect gas leakages in the proximity of the employees that wear it. This is a great way of overcoming the space density problem, as it will monitor the air quality in the near vicinity of the employees, ensuring that they will not find themselves in an environment that might be dangerous for their health. Depending on the gasses that pose the greatest danger, the height at which the device must be placed varies.

Hardware requirements

Nordic Thingy:91
Micro USB cable

Software requirements

Edge Impulse account
Edge Impulse CLI
GIT
nRF Connect 3.11.1
nRF command line tools

Hardware Setup

For this use-case, as mentioned above, we will be using the Thingy:91, a prototyping development kit created by Nordic Semiconductor. It is packed with sensors, making it a great pick for rapid prototyping and also, equipped with a nRF9160 System-in-Package (SiP) that supports LTE-M, NB-IoT and GNSS, allowing you to add a connectivity layer to any application.

This development board comes equipped with a 64 MHz Arm® Cortex®-M33 CPU that is great for running TinyML models on the edge used to detect various phenomena, more specific for our use case, dangerous gas leaks.

Because the board has all the required sensors embedded on it, there is no need for extra wiring. It’s enough to connect the board to a computer and start building the machine learning model.

As for the deployment phase, it comes down to the specific environment in which the system will be deployed. The Thingy:91 weights under 100g so it can be mounted on hard surfaces using regular adhesives or, if the use case allows for it, it can be mounted using screws.

Software Setup

Creating an Edge Impulse Project

To build the machine learning model that will be used to detect dangerous leaks in environments characteristic to the oil and gas industry, we will be using the Edge Impulse platform. Register a free account and create a new project. Remember to give it a representative name and select Something else when asked what kind of data will be used to build the project.

The Nordic Thingy:91 prototyping board is fully supported by the Edge Impulse platform, meaning that you will be able to sample data, build the model and deploy it back on the device straight from the platform, without the need to build any firmware. This is a great time saver because it allows users to optimize their models before having to create custom firmware for the target device

Connecting the Device

To connect the device to Edge Impulse, download nRF connect 3.11.1 and nRF command line tools from the official sources and install them.

If you are going to be using a Linux computer for this application, make sure to run the following command as well:

sudo apt install screen

Afterwards, download the official Edge Impulse Nordic Thingy:91 firmware and extract it.

Next up, make sure the board is turned off and connect it to your computer. Put the board in MCUboot mode by pressing the multi-function button placed in the middle of the device and with the button pressed, turn the board on.

Next, launch the Programmer application in nRF Connect, select your board in the left side of the window, drag and drop the firmware.hex file in the Files area, make sure Enable MCUboot is enabled and press Write.

When prompted with the MCUboot DFU window, press Write and wait for the process to be finished.

If you struggle at any point of this process, Edge Impulse has great documentation on this subject.

Now, power cycle the board by turning it off and on again, this time without pressing the middle button, launch a terminal and issue the following command:

$edge-impulse-daemon

You will be prompted with a message to insert your username and password and then you will be asked to select which device you would like to connect to.

Edge Impulse serial daemon v1.14.10
? What is your user name or e-mail address (edgeimpulse.com)? <your user>
? What is your password? [hidden]

You may notice that the Thingy:91 exposes multiple UARTs. Select the first one and press ENTER.

Edge Impulse serial daemon v1.15.1
Endpoints:
	Websocket: wss://remote-mgmt.edgeimpulse.com
	API:   	https://studio.edgeimpulse.com/v1
	Ingestion: https://ingestion.edgeimpulse.com

? Which device do you want to connect to? /dev/ttyACM0 (Nordic Semiconductor)
[SER] Connecting to /dev/ttyACM0
[SER] Serial is connected, trying to read config...
[SER] Retrieved configuration
[SER] Device is running AT command version 1.3.0

Setting upload host in device... OK
Configuring remote management settings... OK
Configuring API key in device... OK
Configuring HMAC key in device... OK
[SER] Device is not connected to remote management API, will use daemon
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com
[WS ] Device "test" is now connected to project "Oil and Gas Monitoring - Nordic Thingy:91"

By navigating to the Devices tab in your Edge Impulse project, you will notice that your device shows up as connected.

Building the Dataset

To create this data set, we have exposed the gas sensor to various gasses that might be encountered in environments specific to Oil and Gas industries like high concentrations of CO2, CO, and Isopropanol.

To simulate increased concentrations of CO2, we have used a simple set-up that employs baking soda (Sodium Bicarbonate) and vinegar (Acetic Acid). By combining those 2 elements in a confined environment, the CO2 resulting from their reaction, being heavier than air, would spill from the container in which the reaction takes place and get picked up by the Nordic Thingy:91.

To simulate an increased concentration of carbon monoxide, we have exposed the Thingy:91 board to wood smoke. Smoke is fundamentally a complex mixture of fine particles and compounds in a gaseous state like polycyclic aromatic hydrocarbons, nitrogen oxides, sulfur oxides, and carbon monoxide.

Finally, to get a reading specific to an alcohol leakage, we have exposed the gas sensor to a bottle of 97% concentration Isopropanol. Being a very volatile compound, it quickly evaporates and is easily picked up by VOC sensors.

With the device connected, head over to the Data acquisition tab, select Environmental as the sensor that will be used for acquiring data, set the data acquisition frequency to 1Hz and start recording data.

For this application we will be defining 2 classes, labeled “Gas_Leak” and “Normal”. Neural networks do not know what to do with data it has never seen before and will try to classify it in one of the predefined ones. This is why it’s important to define a “Normal” class that contains readings specific to usual conditions in which the system will be deployed, so as to avoid triggering a false positive reading in the detection algorithm.

Keep in mind that machine learning heavily relies on the quantity and quality of data, so when defining a new class make sure to have at least 2.5 minutes of data for it.

With all the data gathered, it’s time to split the data into 2 categories: Training and Testing data.

Edge impulse has automated this process and all you have to do is press the exclamation mark near the Train/Test split section. What we are aiming for is 80% Training data and 20% Testing data.

The dataset used to create this model can be downloaded here.

Designing the Impulse

It's time to design the Impulse now that the dataset is available.

The process of taking raw data from the dataset, pre-processing it into manageable chunks called "windows," extracting the relevant features from them using digital signal processing algorithms, and then feeding them into a classification neural network to determine whether any anomaly in the running regime of the machinery is detected is set up at this point.

An “Impulse” contains all the processes mentioned above in a manageable and easily configurable structure.

To create a gas sensing model, we will be using a 2000ms Window Size, with a Window Increase of 1000ms and a sampling frequency of 1Hz. This will be passed through a Raw Data processing block with only the “gas res” dimensions checked and then, fed into a Classification Neural Network.

Configuring the Raw Data Block

Once you click “Save impulse”, you will notice that every block can be configured by clicking on its name, under the “Impulse Design” submenu.

The Raw data block may be the simplest of the processing blocks, as it has only one parameter that can be modified, namely the “Scale axes” that we will set to 10. On the top side of the screen you can see the time-domain representation of the selected sample.

When you are done configuring the block and exploring the dataset, click on Save parameters.

There are multiple parameters that can be configured in the NN classifier tab that will influence the training process of the classifier neural network. Before configuring those, it is worth understanding how this training process works. Fundamentally, a random value between 0 and 1 is initially assigned to the weight of a link between neurons. Once the training process starts, the neural network is fed the Training dataset defined in the data acquisition phase and the classification output is compared to the correct results. The algorithm then adjusts the weights assigned to the links between neurons and then compares the results once more. This process is repeated for a number of epochs, defined by the Number of training cycles parameter and the Learning rate defines how much the weights are varied each epoch.

During this training process, the neural network may become overtrained and will start to pick up measurement artifacts in the data as the defining feature of a class, instead of looking for the underlying patterns in the data. This process is called overfitting and it's the reason why the performance of a neural network must be evaluated on real world data.

When you click “Start training”, the process will be assigned to a cluster and once the computation ends, you will be presented with the performance of the Neural Network, tested on a percent of samples from the Training dataset held over for validation purposes. The dimension of this datapool is defined by the Validation set size parameter.

When building a machine learning model, what we aim for is a high Accuracy and a low Loss. Accuracy is the percentage of predictions for a given sample where the predicted value coincides with the actual value, and Loss is the total of all errors made for all samples in the validation set.

The confusion matrix presents the percent of samples that were miscategorized and the Data explorer offers a visual representation of the classified samples. It is clearly noticeable that the amount of “Normal” samples wrongly classified as Gas_Leak is greater than the number of Gas_Leak data points categorized as “Normal”. For this particular application, this is not a problem because the model will lean towards triggering a false positive and warn the user that there might be a problem, rather than passing a real gas leak as normal environmental conditions.

Model Testing

The Model Testing tab allows the user to see how the neural network fares when presented with data it has not seen before. Navigate to this tab and click Classify All. Edge Impulse will then feed all the data in the Testing data pool to the neural network and present you with the classification results and the performance of the model, just like during the training process.

Increasing the Model Performance

If your model manifests low performance when met with unseen data, there are various things you can do. First and foremost, the best thing you can do to increase the performance of the model is to give it more data. Neural Networks need a plentiful and balanced data set to be properly trained.

In our case, the dataset was large but we managed to increase the performance of the model by increasing the number of training cycles and the learning rate of the model, a sign that our neural network was not trained enough.

By increasing the number of training cycles to 100 and the Learning rate to 0.001, we observed a jump in performance on unseen data from 95.83% performance to 100% performance.

Deployment

Finally, it’s time to see how to model fares when deployed on the edge. Navigate to the Deployment tab and select Nordic thingy:91 under the Build firmware tab.

After you click on Nordic Thingy:91, you will be presented with the option of enabling the EON Compiler. It’s worth comparing the difference in the resources used when the compiler is turned on vs when it's disabled. Take in consideration the fact that it might increase on-device performance with the price of reduced accuracy. This is very helpful when deploying models on resource-constrained devices or when battery life is an issue but it may not be worth using when the device has plentiful resources and is affixed to a wall with a power source available.

Once you have decided, click on Build.

Once the process ends, deploy the firmware built by the Edge Impulse platform in the same way you have uploaded the data-forwarder firmware during the “Connecting the device” section.

Afterwards, with the board connected to the computer and turned on, issue the following command in a terminal:

$edge-impulse-run-impulse

You will be prompted with the results of the classification that is currently running offline on the device.

Inferencing settings:
    Interval: 1000ms.
    Frame size: 2
    Sample length: 2000ms.
    No. of classes: 2
Starting inferencing, press 'b' to break
Starting inferencing in 2 seconds...
Predictions (DSP: 0 ms., Classification: 2 ms., Anomaly: 0 ms.):
	Gas_Leak:     0.10937
	Normal:     0.89062

Creating Custom Firmware

When you consider the performance of the model when running on the target to be satisfying, Edge Impulse offers its users the ability to export the Impulse as a C++ library that contains signal processing blocks, learning blocks, configurations and the SDK needed to integrate the ML model in a custom application. By choosing this method of deployment you can build applications that can trigger alarms, log data or send notifications remotely by using the connectivity layer provided by the Nordic Thingy:91 platform.

You can find a great guide about how you can Build an application locally for a Zephyr-based Nordic Semiconductor development board in the official Edge Impulse Documentation.

Conclusion

In conclusion, a system built around the Nordic Thingy:91 and Edge Impulse platform is a great way to overcome some of the challenges associated with traditional gas detection systems. By using it as an edge device, it can measure the gas concentration, run a machine learning algorithm and decide if there is a trend in the quantity of dangerous gasses in the air and act out without human intervention. Additionally, by using it as a wearable, it will be able to detect gas leakages in the proximity of the employees that wear it, ensuring that they are not exposed to dangerous levels of gas.

If you need assistance in deploying your own solutions or more information about the tutorial above please reach out to us!

Smart HVAC System with a Sony Spresense

Use a computer vision model running on a Sony Spresense to determine occupancy in rooms, and adapt HVAC zone output accordingly.

Created By: Jallson Suryo

Public Project Link:

GitHub Repo:

Introduction

A common problem found in HVAC systems is that energy is wasted, because the system uses more energy than necessary, or the system cannot quickly adjust to the changing needs in a dynamic environment. To tackle the problem, we need a system that manages its power intensity based on what is necessary for each zone in real-time for a given environment. The power intensity necessary for each zone can be derived from the following data: number of people, time or duration spent inside, and/or the person's activity.

Our Solution

To overcome this challenge, a Smart HVAC System that can optimize energy consumption by adjusting the power intensity in different zones inside an office or a residential space (zones with more people, more activity, and longer time durations will need more cooling/heating and vice versa) could be created. The zone heat mapping will be generated using data obtained from a Sony Spresense microcontroller (with Edge Impulse's FOMO Machine Learning model embedded) that's mounted like a surveillance camera inside the space.

Description

The project uses Edge Impulse's FOMO to detect multiple objects and its coordinates using a compact microcontroller with an on-board camera (the Sony Spresense). The object detection ML model will use the top view of miniature figures with standing and sitting positions as objects. The data captured will be divided into Training and Test data. Then the Impulse with Image and Object Detection as learning blocks and grayscale color blocks will be created.

The accuracy result for this training and test model is above 90%, so there is a higher degree of confidence when counting the number of objects (persons) and tracking their centroid coordinates.

The ML model is then deployed to the Spresense. The number of objects in each zone is displayed on an OLED display. The Spresense also communicates to an Arduino Nano via I2C which we are using for the fan speed controller.

This system will increase fan intensity on areas/zone that need more cooling/heating, which means more activity/people in a certain zone will increase fan intensity in that zone. The total HVAC power output can also be adjusted based on the total number of people in the space.

The project is a proof of concept (PoC) using a 1:50 scale model with an office interior with several partitions and furniture and miniature figures. The space is divided into 4 zones, and each zone has a small fan installed. The OLED display is used in this PoC to show the output of this simulation.

Sony Spresense, aluminium extrusion frame, and 3D printed miniature model (1:50)

System Diagram, prototyping in breadboard, and Smart HVAC System with custom design PCB

Hardware Components:

Sony Spresense
Arduino Nano
2x TB6612 Motor drivers
4x DC 5V mini fan 3cm
0.96inch OLED display
Aluminium extrusion as a camera stand
3D printed (case, office interior 1:50 miniature)
Powerbank & battery for Sony Spresense

Software & Online Services:

Edge Impulse Studio
Arduino IDE

Steps

1. Prepare Data / Photos

In this project we will use a smartphone camera to capture the images for data collection for ease of use. Take pictures from above in different positions with backgrounds of varying angles and lighting conditions to ensure that the model can work under slightly different conditions (to prevent overfitting). Lighting and object size are crucial aspects to ensure the performance of this model.

Note: Keep the size of the objects similar in size in the pictures. Significant difference in object size will confuse the FOMO algorithm.

2. Data Acquisition and Labelling

Choose the Images project option, then Classify Multiple Objects. In Dashboard > Project Info, choose Bounding Boxes for labelling method and Sony Spresense for target device. Then in Data acquisition, click on Upload Data tab, choose your photo files, auto split, then click Begin upload.

Click on Labelling queue tab then start drag a box around an object and label it (person) and Save. Repeat until all images are labelled.

Make sure that the ratio between Training and Test data is ideal, around 80/20.

3. Training and Building Model using FOMO Object Detection

Once you have the dataset ready, go to Create Impulse and set 96 x 96 as the image width - height (this helps in keeping the ML model small). Then choose Fit shortest axis, and choose Image and Object Detection as learning and processing blocks.

Go to Image parameter section, select color depth as Grayscale, then press Save parameters. Then click on Generate and navigate to Object Detection section, and leave training settings for the Neural Network as it is — in our case the defaults work quite well, then we choose FOMO (MobileNet V2 0.35). Train the model by pressing the Start training button. You can see the progress on the right side.

If everything is OK, then we can test the model. Go to Model Testing on the left, then click Classify all. Our result is above 90%, then we can move on to the next step -- deployment.

4. Deploy and Build an Arduino Program

The .ino code is a modified version of the Edge Impulse example code for object detection on Spresense. The modification adds capability to display person count on each room to the OLED screen and act as the controller to the Arduino Nano I2C peripheral. The code distinguishes the four rooms using four quadrants and by knowing the X, Y coordinates of the object’s centroid we can locate the person. The Arduino Nano adjusts the fan motor using PWM based on the number of persons present in the room.

Here is a quick prototype video showing the project:

Conclusion

Finally, we have successfully implemented this object detection model on an Sony Spresense, and use the data captured from the camera to automatically control the HVAC system's fan power intensity and display the occupancy number and power meter for each zone. I believe this Proof of Concept project can be implemented in a real-world HVAC system, so that the goal of optimizing room temperature and saving energy can be achieved for a better, more sustainable future.

Resources

Smart HVAC System with an Arduino Nicla Vision

Use a computer vision model to determine occupancy in rooms and adapt HVAC zone output accordingly.

Created By: Jallson Suryo

Public Project Link: https://studio.edgeimpulse.com/public/215243/latest

GitHub Repo: https://github.com/Jallson/Smart_HVAC

Introduction

Our Solution

To overcome this challenge, a Smart HVAC System that can optimize energy consumption by adjusting the power intensity in different zones inside an office or a residential space (zones with more people, more activity, and longer time durations will need more cooling/heating and vice versa) could be created. The zone heat mapping will be generated using data obtained from an Arduino Nicla Vision (with Edge Impulse's FOMO Machine Learning model embedded) that's mounted like a surveillance camera inside the space.

Description

The project uses Edge Impulse's FOMO to detect multiple objects and its coordinates using a compact micro-controller with an on-board camera (the Nicla Vision). The object detection ML model will use the top view of miniature figures with standing and sitting positions as objects. The data captured will be divided into Training and Test data. Then the Impulse with Image and Object Detection as learning blocks and grayscale color blocks will be created.

The accuracy result for this training and test model is above 90%, so there is a higher degree of confidence when counting the number of objects (persons) and tracking their centroid coordinates.

The ML model is then deployed to the Nicla Vision. The number of objects in each zone is displayed on an OLED display. The Nicla Vision also communicates to an Arduino Nano via I2C which we are using for the fan speed controller.

Arduino Nicla Vision, aluminium extrusion frame, and 3D printed miniature model (1:50)

System Diagram, prototyping in breadboard, and Smart HVAC System with custom design PCB

Hardware Components:

Arduino Nicla Vision
Arduino Nano
2x TB6612 Motor drivers
4x DC 5V mini fan 3cm
0.96inch OLED display
Aluminium extrusion as a camera stand
3D printed (case, office interior 1:50 miniature)
Powerbank & battery for Nicla Vision

Software & Online Services:

Edge Impulse Studio
Arduino IDE
OpenMV IDE

Steps

1. Prepare Data / Photos

Note: Keep the size of the objects similar in size in the pictures. Significant difference in object size will confuse the FOMO algorithm.

2. Data Acquisition and Labelling

Open studio.edgeimpulse.com, login (or create an account first), then create a new Project.

Choose the Images project option, then Classify Multiple Objects. In Dashboard > Project Info, choose Bounding Boxes for labelling method and Nicla Vision for target device. Then in Data acquisition, click on Upload Data tab, choose your photo files, auto split, then click Begin upload.

Click on Labelling queue tab then start drag a box around an object and label it (person) and Save. Repeat until all images are labelled.

Make sure that the ratio between Training and Test data is ideal, around 80/20.

3. Training and Building Model using FOMO Object Detection

Once you have the dataset ready, go to Create Impulse and set 96 x 96 as the image width - height (this helps in keeping the ML model small, to fit within the Nicla's memory size). Then choose Fit shortest axis, and choose Image and Object Detection as learning and processing blocks.

If everything is OK, then we can test the model. Go to Model Testing on the left, then click Classify all. Our result is above 90%, then we can move on to the next step -- deployment.

4. Deploy to OpenMV Firmware and Test

To use the OpenMV firmware, you will need the OpenMV IDE installed on your computer. Once you have the IDE ready, if you check the downloaded .zip folder you will find a number of files. We will need the following files:

edge_impulse_firmware_arduino_nicla_vision.bin and ei_object_detection.py.

The next step is loading the downloaded firmware containing the ML model to the Nicla Vision board. So go back to OpenMV and go to Tools -> Run Bootloader (Load Firmware), select the .bin file in the unzipped folder, and click Run.

Next, we will run the python code. Go to File -> Open File and select the .py python file from the unzipped folder. Once the file is opened, connect the Nicla Vision board, select the serial/com port and click the green “play” button. The program should be running now and you can see the FOMO object detection will run in a small window.

5. Deploy and Build an Arduino Program

You should have the Arduino IDE installed on your computer for the following step. Once the Edge Impulse Arduino Firmware is built, downloaded and unzipped, you should download the nicla_vision_camera_smartHVAC_oled.ino code which can be downloaded here and place it inside the unzipped folder from Edge Impulse. Once the .ino code is inside Edge Impulse unzipped folder, move it to your Arduino folder on your computer. Now you can upload the .ino code to your Nicla Vision board via the Arduino IDE.

The .ino code is a modified version of the Edge Impulse example code for object detection on Nicla Vision. The modification adds capability to display person count on each room to the OLED screen and act as the controller to the Arduino Nano I2C peripheral. The code distinguishes the four rooms using four quadrants and by knowing the X, Y coordinates of the object’s centroid we can locate the person. The Arduino Nano adjusts the fan motor using PWM based on the number of persons present in the room.

The code for the Arduino Nano peripheral Nano_SmartHVAC_I2C_Peripheral.ino can be downloaded here.

Here is a quick prototype video showing the project:

Conclusion

Finally, we have successfully implemented this object detection model on an Arduino Nicla Vision, and use the data captured from the camera to automatically control the HVAC system's fan power intensity and display the occupancy number and power meter for each zone. I believe this Proof of Concept project can be implemented in a real-world HVAC system, so that the goal of optimizing room temperature and saving energy can be achieved for a better, more sustainable future.

Complete demo video showing the project:

Indoor CO2 Level Estimation - Arduino Portenta H7

Estimate the CO2 level in an indoor environment by counting the people in the room using TinyML.

Created By: Swapnil Verma

Public Project Link:

Problem Overview

It has been almost two and half years since the COVID-19 pandemic started. After multiple vaccines and numerous tests, we are slowly going back to our old life; for me, it's going back to the office, seeing people and organising face to face meetings (along with video calls, of course). Even though we are going back to our old lives, COVID is far from over, and to prevent and monitor infection, we have specific arrangements in place. One such arrangement is CO2 monitors in an indoor environment. One study suggests that we can predict the infection risk by observing a CO2 level in an indoor environment[1]. A higher level of CO2 means poor ventilation and/or higher occupancy, thus a higher infection risk.

Can we predict a higher infection risk using any other technique? Let us explore our options.

Solution

My solution uses a TinyML based algorithm to detect and count the people in an indoor environment. The algorithm will be deployed on a microcontroller. The microcontroller will capture an image or stream of images using a camera and then perform inference on the device to count people.

The device can record the occupancy level locally or send it to a remote machine, possibly a server, for further evaluation. After counting the number of people in an indoor environment, we can do all sorts of things. For example, we can calculate the approximate CO2 level in the room, the distance b/w people to predict the infection risk [2], etc. In this project, I will focus on CO2 level estimation.

Hardware

The hardware I am proposing for this project is pretty simple. It consists of

An Arduino Portenta H7
An Arduino Portenta Vision Shield

Software

1. Dataset

In this project, the dataset I am using is a subset of the PIROPO database [3].

PIROPO Database - https://sites.google.com/site/piropodatabase/

The dataset contains multiple sequences recorded in the two indoor rooms using a perspective camera.

The original PIROPO database contains perspective as well as omnidirectional camera images.

I used this feature to label people in the PIROPO images. I then divided the data into training and test sets using the train/test split feature. While training, the Edge Impulse automatically divides the training dataset into training and validation datasets.

2. Training and Testing

The training F1 score of my model is 91.6%, and the testing accuracy is 86.42%. For live testing, I deployed the model by building openMV firmware and flashed that firmware using the OpenMV IDE. A video of live testing performed on Arduino Portenta H7 is attached in the Demo section below.

3. How to run the software?

This section contains a step-by-step guide to downloading and running the software on the Arduino Portenta H7.

Open the ei_object_detection.py and run it in the OpenMV IDE.

How does it work?

This system is quite simple. The Vision shield (or any camera) captures a 240x240 image of the environment and passes it to the FOMO model prepared using Edge Impulse. This model then identifies the people in the image and passes the number of people to the CO2 level estimation function every minute. The function then estimates the amount of CO2 using the below formula.

The average human exhales about 2.3 pounds of carbon dioxide on an average day [4], and the magic number 0.02556 comes by dividing 2.3 by 24x60 (minutes in a day) and converting it into ounces. The equation calculates the amount of CO2 in ounces per minute. The person detection model can be used with any other application for example occupancy detection etc. The system then repeats this process again.

Demos

The testing accuracy of this model is 86.4% when tested with the PIROPO dataset. But that is not the final test of this model. It should perform well when introduced to a new environment, and that is exactly what I did. I used this model on the Arduino Portenta H7 with the vision shield to detect myself in my living room. The model has never seen me nor my living room before. Let's see how well it performs.

The calculation of CO2 level is a straightforward task compared to person detection, therefore, in these demos, I have focused only on person detection.

Note: These videos are 240x240 in resolution and are recorded using the OpenMV IDE. The FPS (Frame Per Second) improves when the device is not connected to the OpenMV IDE.

In the above video, the model is doing quite well. It is missing me in some frames but it has got a lock on me most of the time. I was surprised to see it work this well even when I was behind the couch.

In the above test, I am testing how well it detects someone standing far from the camera. To my surprise, it is detecting me well even when I am standing farthest I could be in this room. It is also detecting me when I am a little sideways while walking.

In the above test, I wanted to see the system's performance when I am sitting on a chair. It works excellent when it sees me from my side but it is not detecting me when I am facing the camera. I think it is because, in my training data, all samples where a person is sitting on a chair capture their side profile and not the front profile. It can be improved by using datasets which contain a person sitting on a chair and facing toward the camera.

Conclusion

Looking at the live testing performance of this model, it is clear that the model is working quite well but has some room for improvement. Considering that the inference is performed on a Microcontroller with 240x240 image data, I am happy with the results. As a next step, I will try to improve its capability as well as accuracy by using diverse training data.

The CO2 level estimation is a simple task given the person detection model has good accuracy and repeatability. The next step for this application would be to improve the estimation by also considering the flow of CO2 out of an indoor space.

Reference

Harmful Gases Detection - Arduino Nano 33 BLE Sense

Detect harmful gases with machine learning and an Arduino Nano 33 BLE plus gas sensor.

Created By: Roni Bandini

Public Project Link:

Project Demo

Intro

Industries working with chemicals are always subject to leaks that could harm workers. A Machine Learning model could be trained to identify subtle relationships between multiple gas readings to spot custom leaks.

Hardware Used

Arduino BLE 33 Sense
MiCS-4514 gas sensor
5V Cooler
Oled screen 128x32
Buzzer
3d printed parts
DC female connector
5v Power Supply

This project uses MiCS-4514 multi gas sensor. The sensor is able to detect the following gases:

Methane (CH4) (1000–25000) PPM
Ethanol (C2H5OH) (10-500) PPM
Hydrogen (H2) (1-1000) PPM
Ammonia (NH3) (1-500) PPM
Carbon Monoxide (CO) (1- 1000) PPM
Nitrogen Dioxide (NO2) (0.1-10) PPM

The device will read the gas sensor x times for y seconds and then calculate min, max and average. Those values will be forwarded to the Machine Learning model for inference and then a score will be obtained for “harmful” gas 1 or “regular” gas 2.

Data Acquisition

Upload to the Arduino BLE 33 Sense the acquisition script. Place the sensor unit close to the gas or substance and open serial monitor. You should see there multi gas sensor values as CSV records. Uncheck timestamp, then copy and paste the serial monitor screen into a text file. Save that file as harmful.csv and add this header:

timestamp, CO2avg, C2H5OHavg, H2avg, NH3avg, CO2min, C2H5OHmin, H2min, NH3min, CO2max, C2H5OHmax, H2max, NH3max

Repeat the procedure for all the gases to be detected.

Model Training

Go to Impulse Design, Create Impulse.

In Times Series Data, use 1500ms Windows Size and 0,6 frequency.

Processing block will be raw data with all axis checked. For classification use Keras with 2 output features: regular and harmful.

In Raw Data you can see all values for regular and harmful inside every windows size. Then you have to click Generate Features.

For NN Classifier use 60 training cycles, 0.0005 Learning Rate, Validation 20 and Autobalance dataset. Add an extra layer Dropout Rate 0.1 Click Start Training and check if you get good accuracy.

If you are ok with results you can go to Model Testing and check the performance with new data. If there are lots of readings with wrong classification you should check again data acquisition procedure.

Deployment

Go to Deployment. Select Arduino Library and save the zip file. Now go to Arduino IDE, Sketch, Include Library, Add Zip library and select the downloaded zip file.

Connect the Arduino to the computer with the USB cable and Upload the code. Please note that it will take long minutes to upload.

Code Settings

If you want to change calibration time in minutes, use #define CALIBRATION_TIME 3

If your OLED screen has a different I2C address, use #define SCREEN_ADDRESS 0x3C

If you want to use another pin for the Buzzer # define pinBuzzer 2

If you want to use more readings for min, max and average int measuresNumber=4;

If you want to change the measure timeframe int measuresTimeFrame=1500; // 1.5 seconds

If you want to change the percentage to identify the gases float scoreLimit=0.8;

Using Bhopal 84

After displaying Bhopal and Edge Impulse logos, the unit will start the calibration phase. During this phase do not place any substance or gas under the sensor unit. It should read normal air conditions. As soon as this step is finished you can put the leaked substance under the sensor unit and it should be detected in 1.5 seconds. Why 1.5 seconds? During that period 4 readings will be made to obtain mix, max and average for all gases. That information will be forwarded to the model and a classification will be returned.

The prototype is able to detect normal air conditions, a regular gas and a harmful gas.

Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense

Demonstrating Sensor Fusion on an Arduino Nano 33 BLE Sense to detect the presence of a fire using TinyML.

Created By: Nekhil R.

Public Project Link:

Story

In order to properly identify a fire situation, a fire detection system needs to be accurate and fast. However, many commercial fire detection systems use simple sensors, so their fire recognition accuracy can be reduced due to the limitations of the sensor's detection capabilities. Existing devices that use rule-based algorithms or image-based machine learning might be unable to adapt to changes in the environment because of their static features.

In this project, we will develop a device that can detect fire by means of sensor fusion and machine learning. The combination of sensors will help to make more accurate predictions about the presence of fire, versus single-sensor monitoring. We will collect data from sensors such as temperature, humidity, and pressure in various fire situations and extract features to build a machine-learning model to detect fire events.

Data Collection

To make this project a reality, we are using an Arduino Nano 33 BLE Sense with Edge Impulse. For data collection, there are two ways to get data samples into the Edge Impulse platform: either through the Edge Impulse CLI, or through a web browser logged into the Edge Impulse Studio.

Collecting data through the web browser is simple and straightforward. To do this, connect the device to your computer and open the Edge Impulse Studio. Press the Connect Using WebUSB button, and select your development board. The limitation of using the web serial integration is that it only works with development boards that have full Edge Impulse support.

The data collection settings for our project are shown below. Using temperature, humidity and pressure as environmental sensors is a good choice, as they are the parameters that change the most in case of fire events. Also, the sampling rate of 12.5 Hz is appropriate as these parameters are slow-moving.

We have only two classes in this project: No Fire and Fire. For the No Fire case, we collected data at different points in the room. For capturing the Fire data, we built a fire using a camp-like setup in my backyard. To make our model robust, we collected data at different points in the area.

13 minutes of data are collected for two labels and split between Training and Testing datasets. Once the data is uploaded, Edge Impulse has a tool called Data Explorer which gives you a graphical overview of your complete dataset.

This tool is very useful for quickly looking for outliers and discrepancies in your labels and data points.

Impulse Design

This is our machine learning pipeline, known as an Impulse

These are our Spectral Analysis parameters of Filter and Spectral power. We didn't use any filter for the raw data.

The below image shows the Generated features for the collected data, and we can see that the data is well separated and distinguishable. As you can notice in the case of Fire event, there are actually three clusters of data and it shows the parameters are changing at different points.

Model Training

After successfully extracting the features from the DSP block, it's time to train the machine learning model.

Here are our Neural Network settings and architecture, which work very well for our data.

After training, we achieved 98% validation accuracy for the data, so the model seems to be working well.

The Confusion matrix is a great tool for evaluating the model, as you can see below, 2.1% of the data samples are misclassified as No Fire.

By checking the Feature explorer we can easily identify the samples which are misclassified. It also shows the time at which the incorrect classification happened. Here is one example.

This machine learning model seems to be working well enough for our project, so let's see how our model performs on unseen data.

Model Testing

When collecting data, some data is set aside and not used in the Training process. This is called Test data, and we can use it now to check how our model performs on unseen data.

The Confusion matrix and Feature explorer show that our model performs very well.

Live Classification

Now let's test the model with some real-world data. For that we need to move onto the Live Classification tab and connect our Arduino using WebUSB once again.

The above sample was recorded the when there is no fire present, and the below sample is recorded when there is a fire.

It looks like real-world data of No Fire and Fire events are well classified, so our model is ready for deployment onto the Arduino.

Deployment

Push Notification

Hardware

The complete hardware unit consists of the Arduino Nano 33 BLE Sense, power adapter, and an ESP-01. The ESP-01 is used to add WiFi connectivity to the Arduino. This component handles sending email alerts over a designated WiFi connection. This occurs via serial communication between the Arduino and ESP-01. In order to establish this communication, we first need to upload the necessary code to both the ESP-01 and the Arduino, which can be found in the GitHub repository linked below. Afterwards, we connected the components according to this schematic:

This is the final hardware setup for the project:

Code

AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3

Use a XIAO ESP32C3 to monitor temperature, humidity, and pressure to help aid in dairy manufacturing processes.

Created By: Kutluhan Aktar

Public Project Link:

Description

As many of us know, yogurt is produced by bacterial fermentation of milk, which can be of cow, goat, ewe, sheep, etc. The fermentation process thickens the milk and provides a characteristic tangy flavor to yogurt. Considering organisms contained in yogurt stimulate the gut's friendly bacteria and suppress harmful bacteria looming in the digestive system, it is not surprising that yogurt is consumed worldwide as a healthy and nutritious food[^1].

The bacteria utilized to produce yogurt are known as yogurt cultures (or starters). Fermentation of sugars in the milk by yogurt cultures yields lactic acid, which decomposes and coagulates proteins in the milk to give yogurt its texture and characteristic tangy flavor. Also, this process improves the digestibility of proteins in the milk and enhances the nutritional value of proteins. After the fermentation of the milk, yogurt culture could help the human intestinal tract to absorb the amino acids more efficiently[^2].

Even though yogurt production and manufacturing look like a simple task, achieving precise yogurt texture (consistency) can be arduous and strenuous since various factors affect the fermentation process while processing yogurt, such as:

Temperature
Humidity
Pressure
Milk Temperature
Yogurt Culture (Starter) Amount (Weight)

In this regard, most companies employ food (chemical) additives while mass-producing yogurt to maintain its freshness, taste, texture, and appearance. Depending on the production method, yogurt additives can include dilutents, water, artificial flavorings, rehashed starch, sugar, and gelatine.

In recent years, due to the surge in food awareness and apposite health regulations, companies were coerced into changing their yogurt production methods or labeling them conspicuously on the packaging. Since people started to have a penchant for consuming more healthy and organic (natural) yogurt, it became a necessity to prepare prerequisites precisely for yogurt production without any additives. However, unfortunately, organic (natural) yogurt production besets some local dairies since following strict requirements can be expensive and demanding for small businesses trying to gain a foothold in the dairy industry.

After perusing recent research papers on yogurt production, I decided to utilize temperature, humidity, pressure, milk temperature, and culture weight measurements denoting yogurt consistency before fermentation so as to create an easy-to-use and budget-friendly device in the hope of assisting dairies in reducing total cost and improving product quality.

Even though the mentioned factors can provide insight into detecting yogurt consistency before fermentation, it is not possible to extrapolate and construe yogurt texture levels precisely by merely employing limited data without applying complex algorithms. Hence, I decided to build and train an artificial neural network model by utilizing the empirically assigned yogurt consistency classes to predict yogurt texture levels before fermentation based on temperature, humidity, pressure, milk temperature, and culture weight measurements.

Since XIAO ESP32C3 is an ultra-small size IoT development board that can easily collect data and run my neural network model after being trained to predict yogurt consistency levels, I decided to employ XIAO ESP32C3 in this project. To collect the required measurements to train my model, I used a temperature & humidity sensor (Grove), an integrated pressure sensor kit (Grove), an I2C weight sensor kit (Gravity), and a DS18B20 waterproof temperature sensor. Since the XIAO expansion board provides various prototyping options and built-in peripherals, such as an SSD1306 OLED display and a MicroSD card module, I used the expansion board to make rigid connections between XIAO ESP32C3 and the sensors.

Since the expansion board supports reading and writing information from/to files on an SD card, I stored the collected data in a CSV file on the SD card to create a data set. In this regard, I was able to save data records via XIAO ESP32C3 without requiring any additional procedures.

After completing my data set, I built my artificial neural network model (ANN) with Edge Impulse to make predictions on yogurt consistency levels (classes). Since Edge Impulse is nearly compatible with all microcontrollers and development boards, I had not encountered any issues while uploading and running my model on XIAO ESP32C3. As labels, I utilized the empirically assigned yogurt texture classes for each data record while collecting yogurt processing data:

Thinner
Optimum
Curdling (Lumpy)

After training and testing my neural network model, I deployed and uploaded the model on XIAO ESP32C3. Therefore, the device is capable of detecting precise yogurt consistency levels (classes) by running the model independently.

Since I wanted to allow the user to get updates and control the device remotely, I decided to build a complementing Blynk application for this project: The Blynk dashboard displays the recent sensor readings transferred from XIAO ESP32C3, makes XIAO ESP32C3 run the neural network model, and shows the prediction result.

Lastly, to make the device as sturdy and robust as possible while operating in a dairy, I designed a dairy-themed case with a sliding (removable) front cover (3D printable).

So, this is my project in a nutshell 😃

In the following steps, you can find more detailed information on coding, logging data on the SD card, communicating with a Blynk application, building a neural network model with Edge Impulse, and running it on XIAO ESP32C3.

Step 1: Designing and printing a dairy-themed case

Since I focused on building a budget-friendly and easy-to-use device that collects yogurt processing data and informs the user of the predicted yogurt consistency level before fermentation, I decided to design a robust and sturdy case allowing the user to access the SD card after logging data and weigh yogurt culture (starter) easily. To avoid overexposure to dust and prevent loose wire connections, I added a sliding front cover with a handle to the case. Also, I decided to emboss yogurt and milk icons on the sliding front cover so as to complement the dairy theme gloriously.

Since I needed to adjust the rubber tube length of the integrated pressure sensor, I added a hollow cylinder part to the main case to place the rubber tube. Then, I decided to fasten a small cow figure to the cylinder part because I thought it would make the case design align with the dairy theme.

I designed the main case and its sliding front cover in Autodesk Fusion 360. You can download their STL files below.

For the cow figure (replica) affixed to the top of the cylinder part of the main case, I utilized this model from Thingiverse:

Then, I sliced all 3D models (STL files) in Ultimaker Cura.

Since I wanted to create a solid structure for the main case with the sliding front cover representing dairy products, I utilized these PLA filaments:

Beige
ePLA-Matte Milky White

Finally, I printed all parts (models) with my Creality Sermoon V1 3D Printer and Creality CR-200B 3D Printer in combination with the Creality Sonic Pad. You can find more detailed information regarding the Sonic Pad in Step 1.1.

If you are a maker or hobbyist planning to print your 3D models to create more complex and detailed projects, I highly recommend the Sermoon V1. Since the Sermoon V1 is fully-enclosed, you can print high-resolution 3D models with PLA and ABS filaments. Also, it has a smart filament runout sensor and the resume printing option for power failures.

Furthermore, the Sermoon V1 provides a flexible metal magnetic suction platform on the heated bed. So, you can remove your prints without any struggle. Also, you can feed and remove filaments automatically (one-touch) due to its unique sprite extruder (hot end) design supporting dual-gear feeding. Most importantly, you can level the bed automatically due to its user-friendly and assisted bed leveling function.

Step 1.1: Improving print quality and speed with the Creality Sonic Pad

Creality Sonic Pad is a beginner-friendly device to control almost any FDM 3D printer on the market with the Klipper firmware. Since the Sonic Pad uses precision-oriented algorithms, it provides remarkable results with higher printing speeds. The built-in input shaper function mitigates oscillation during high-speed printing and smooths ringing to maintain high model quality. Also, it supports G-code model preview.

Step 1.2: Assembling the case and making connections & adjustments

As shown in the schematic below, before connecting the DS18B20 waterproof temperature sensor to the expansion board, I attached a 4.7K resistor as a pull-up from the DATA line to the VCC line of the sensor to generate accurate temperature measurements.

To display the collected data, I utilized the built-in SSD1306 OLED screen on the expansion board. To assign yogurt consistency levels empirically while saving data records to a CSV file on the SD card, I used the built-in MicroSD card module and button on the expansion board.

After printing all parts (models), I fastened all components except the expansion board to their corresponding slots on the main case via a hot glue gun.

I attached the expansion board to the main case by utilizing M3 screws with hex nuts and placed the rubber tube of the integrated pressure sensor in the hollow cylinder part of the main case.

Then, I placed the sliding front cover via the dents on the main case.

Finally, I affixed the small cow figure to the top of the cylinder part of the main case via the hot glue gun.

Step 2: Creating a Blynk application and user interface for XIAO ESP32C3

Since I focused on building an accessible device, I decided to create a complementing Blynk application for allowing the user to display recent sensor readings, run the Edge Impulse neural network model, and get informed of the prediction result remotely.

Since Blynk allows the user to adjust the unit, data range, and color scheme for each widget, I was able to create a unique web user interface for the device.

Temperature Gauge ➡ V4
Humidity Gauge ➡ V12
Pressure Gauge ➡ V6
Milk Temperature Gauge ➡ V7
Weight Gauge ➡ V8
Switch Button ➡ V9
Label ➡ V10

After completing designing the web user interface, I tested the virtual pin connection of each widget with XIAO ESP32C3.

Step 3: Setting up XIAO ESP32C3 on the Arduino IDE

Since the XIAO expansion board supports reading and writing information from/to files on an SD card, I decided to log the collected yogurt processing data in a CSV file on the SD card without applying any additional procedures. Also, I employed XIAO ESP32C3 to communicate with the Blynk application to run the neural network model remotely and transmit the collected data.

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

Since the provided XIAO ESP32C3 core's assigned pin numbers are not compatible with the expansion board's MicroSD card module, it throws an error on the Arduino IDE while attempting to access the SD card.

Therefore, I needed to change the assigned SS pin to 4 (GPIO4) in the pins_arduino.h file.

Step 3.1: Displaying images on the SSD1306 OLED screen

To display images (black and white) on the SSD1306 OLED screen successfully, I needed to create monochromatic bitmaps from PNG or JPG files and convert those bitmaps to data arrays.

Step 4: Logging yogurt processing information in a CSV file on the SD card w/ XIAO ESP32C3

After setting up XIAO ESP32C3 and installing the required libraries, I programmed XIAO ESP32C3 to collect environmental factor measurements and the culture (starter) amount in order to save them to the given CSV file on the SD card.

Temperature (°C)
Humidity (%)
Pressure (kPa)
Milk Temperature (°C)
Starter Weight (g)

Since I needed to assign yogurt consistency levels (classes) empirically as labels for each data record while collecting yogurt processing data to create a valid data set for the neural network model, I utilized the built-in button on the XIAO expansion board in two different modes (long press and short press) so as to choose among classes and save data records. After selecting a yogurt consistency level (class) by short-pressing the button, XIAO ESP32C3 appends the selected class and the recently collected data to the given CSV file on the SD card as a new row if the button is long-pressed.

Button (short-pressed) ➡ Select a class (Thinner, Optimum, Curdling)
Button (long-pressed) ➡ Save data to the SD card

You can download the AI_yogurt_processing_data_collect.ino file to try and inspect the code for collecting yogurt processing data and for saving data records to the given CSV file on the SD card.

⭐ Include the required libraries.

⭐ Initialize the File class and define the CSV file name on the SD card.

⭐ Define the 0.96 Inch SSD1306 OLED display on the XIAO expansion board.

⭐ Define the temperature & humidity sensor object (Grove), the I2C weight sensor object (Gravity), and the DS18B20 waterproof temperature sensor settings.

⭐ Define monochrome graphics.

⭐ Define the built-in button pin on the expansion board.

⭐ Then, define the button state and the duration variables to utilize the button in two different modes: long press and short press.

⭐ Initialize the SSD1306 screen.

⭐ Initialize the DS18B20 temperature sensor.

⭐ Define the required settings to initialize the temperature & humidity sensor (Grove).

⭐ In the err_msg function, display the error message on the SSD1306 OLED screen.

⭐ Check the temperature & humidity sensor connection status and print the error message on the serial monitor, if any.

⭐ Check the connection status between the I2C weight sensor and XIAO ESP32C3.

⭐ Set the calibration weight (g) and threshold (g) to calibrate the weight sensor automatically.

⭐ Display the current calibration value on the serial monitor.

⭐ Check the connection status between XIAO ESP32C3 and the SD card.

⭐ In the get_temperature_and_humidity function, obtain the measurements generated by the temperature & humidity sensor.

⭐ In the get_pressure function, get the measurements generated by the integrated pressure sensor (Grove).

⭐ Then, convert the accumulation of raw data to accurate pressure estimation.

⭐ In the get_weight function, obtain the weight measurement generated by the I2C weight sensor.

⭐ Then, subtract the container weight from the total weight to get the net weight.

⭐ In the get_milk_temperature function, obtain the temperature measurement generated by the DS18B20 temperature sensor.

⭐ In the home_screen function, display the collected data and the selected class on the SSD1306 OLED screen.

⭐ In the save_data_to_SD_Card function:

⭐ Open the given CSV file on the SD card in the APPEND file mode.

⭐ If the given CSV file is opened successfully, create a data record from the recently collected data, including the selected yogurt consistency level (class), to be inserted as a new row.

⭐ Then, append the recently created data record and close the CSV file.

⭐ After appending the given data record successfully, notify the user by displaying this message on the SSD1306 OLED screen: Data saved to the SD card!

⭐ Detect whether the built-in button is short-pressed or long-pressed.

⭐ If the button is short-pressed, change the class number [0 - 2] to choose among yogurt consistency levels (classes).

⭐ If the button is long-pressed, append the recently created data record to the given CSV file on the SD card.

Step 4.1: Collecting samples while producing yogurt to create a data set

After uploading and running the code for collecting yogurt processing data and for saving information to the given CSV file on the SD card on XIAO ESP32C3:

🐄🥛📲 The device shows the opening screen if the sensor and MicroSD card module connections with XIAO ESP32C3 are successful.

🐄🥛📲 Then, the device displays the collected yogurt processing data and the selected class number on the SSD1306 OLED screen:

Temperature (°C)
Humidity (%)
Pressure (kPa)
Milk Temperature (°C)
Starter Weight (g)
Selected Class

🐄🥛📲 If the button (built-in) is short-pressed, the device increments the selected class number in the range of 0-2:

Thinner [0]
Optimum [1]
Curdling [2]

🐄🥛📲 If the button (built-in) is long-pressed, the device appends the recently created data record from the collected data to the yogurt_data.csv file on the SD card, including the selected yogurt consistency class number under the consistency_level data field.

🐄🥛📲 After successfully appending the data record, the device notifies the user via the SSD1306 OLED screen.

🐄🥛📲 If XIAO ESP32C3 throws an error while operating, the device shows the error message on the SSD1306 OLED screen and prints the error details on the serial monitor.

🐄🥛📲 Also, the device prints notifications and sensor measurements on the serial monitor for debugging.

To create a data set with eminent validity and veracity, I collected yogurt processing data from nearly 30 different batches. Since I focused on predicting yogurt texture precisely, I always used cow milk in my experiments but changed milk temperature, yogurt culture (starter) amount, and environmental factors while conducting my experiments.

🐄🥛📲 After completing logging the collected data in the yogurt_data.csv file on the SD card, I elicited my data set.

Step 5: Building a neural network model with Edge Impulse

When I completed logging the collected data and assigning labels, I started to work on my artificial neural network model (ANN) to detect yogurt consistency (texture) levels before fermentation so as to improve product quality and reduce the total cost for small dairies.

Since Edge Impulse supports almost every microcontroller and development board due to its model deployment options, I decided to utilize Edge Impulse to build my artificial neural network model. Also, Edge Impulse makes scaling embedded ML applications easier and faster for edge devices such as XIAO ESP32C3.

Even though Edge Impulse supports CSV files to upload samples, the data type should be time series to upload all data records in a single file. Therefore, I needed to follow the steps below to format my data set so as to train my model accurately:

Data Scaling (Normalizing)
Data Preprocessing

As explained in the previous steps, I assigned yogurt consistency classes empirically while logging yogurt processing data from various batches. Then, I developed a Python application to scale (normalize) and preprocess data records to create appropriately formatted samples (single CSV files) for Edge Impulse.

Since the assigned classes are stored under the consistency_level data field in the yogurt_data.csv file, I preprocessed my data set effortlessly to create samples from data records under these labels:

0 — Thinner
1 — Optimum
2 — Curdling

Plausibly, Edge Impulse allows building predictive models optimized in size and accuracy automatically and deploying the trained model as an Arduino library. Therefore, after scaling (normalizing) and preprocessing my data set to create samples, I was able to build an accurate neural network model to predict yogurt consistency levels and run it on XIAO ESP32C3 effortlessly.

Step 5.1: Preprocessing and scaling the data set to create formatted samples for Edge Impulse

If the data type is not time series, Edge Impulse cannot distinguish data records as individual samples from one CSV file while adding existing data to an Edge Impulse project. Therefore, the user needs to create a separate CSV file for each sample, including a header defining data fields.

To scale (normalize) and preprocess my data set so as to create individual CSV files as samples automatically, I developed a Python application consisting of one file:

process_dataset.csv

Since Edge Impulse can infer the uploaded sample's label from its file name, the application reads the given CSV file (data set) and generates a separate CSV file for each data record, named according to its assigned yogurt consistency class number under the consistency_level data field. Also, the application adds a sample number incremented by 1 for generated CSV files sharing the same label:

Thinner.sample_1.csv
Thinner.sample_2.csv
Optimum.sample_1.csv
Optimum.sample_2.csv
Curdling.sample_1.csv
Curdling.sample_2.csv

First of all, I created a class named process_dataset in the process_dataset.py file to bundle the following functions under a specific structure.

⭐ Include the required modules.

⭐ In the init function, read the data set from the given CSV file and define the yogurt consistency class names.

⭐ In the scale_data_elements function, scale (normalize) data elements to define appropriately formatted data items in the range of 0-1.

⭐ In the split_dataset_by_labels function:

⭐ Split data records by the assigned yogurt consistency level (class).

⭐ Add the header defining data fields as the first row.

⭐ Create scaled data records with the scaled data elements and increase the sample number for each scaled data record sharing the same label.

⭐ Then, generate CSV files (samples) from scaled data records, named with the assigned yogurt consistency level and the given sample number.

⭐ Each sample includes five data items [shape=(5,)]:

*[0.2304, 0.7387, 0.34587, 0.4251, 0.421] *

temperature
humidity
pressure
milk_temperature
starter_weight

⭐ Finally, create appropriately formatted samples as individual CSV files and save them in the data folder.

🐄🥛📲 After running the application, it creates samples, saves them under the data folder, and prints generated CSV file names on the shell for debugging.

Step 5.2: Uploading formatted samples to Edge Impulse

After generating training and testing samples successfully, I uploaded them to my project on Edge Impulse.

Step 5.3: Training the model on yogurt consistency levels

After uploading my training and testing samples successfully, I designed an impulse and trained it on yogurt consistency levels (classes).

An impulse is a custom neural network model in Edge Impulse. I created my impulse by employing the Raw Data processing block and the Classification learning block.

The Raw Data processing block generate windows from data samples without any specific signal processing.

The Classification learning block represents a Keras neural network model. Also, it lets the user change the model settings, architecture, and layers.

According to my experiments with my neural network model, I modified the neural network settings and layers to build a neural network model with high accuracy and validity:

📌 Neural network settings:

Number of training cycles ➡ 50
Learning level ➡ 0.005
Validation set size ➡ 20

📌 Extra layers:

Dense layer (20 neurons)
Dense layer (10 neurons)

After generating features and training my model with training samples, Edge Impulse evaluated the precision score (accuracy) as 100%.

The precision score (accuracy) is approximately 100% due to the modest volume and variety of training samples from different batches. In technical terms, the model trains on limited validation samples. Therefore, I am still collecting data to improve my training data set.

Step 5.4: Evaluating the model accuracy and deploying the model

After building and training my neural network model, I tested its accuracy and validity by utilizing testing samples.

The evaluated accuracy of the model is 100%.

After validating my neural network model, I deployed it as a fully optimized and customizable Arduino library.

Step 6: Setting up the Edge Impulse model on XIAO ESP32C3

After building, training, and deploying my model as an Arduino library on Edge Impulse, I needed to upload the generated Arduino library on XIAO ESP32C3 to run the model directly so as to create an easy-to-use and capable device operating with minimal latency, memory usage, and power consumption.

Since Edge Impulse optimizes and formats signal processing, configuration, and learning blocks into a single package while deploying models as Arduino libraries, I was able to import my model effortlessly to run inferences.

After importing my model successfully to the Arduino IDE, I programmed XIAO ESP32C3 to run inferences when the switch button on the Blynk web application is activated so as to detect yogurt consistency (texture) levels before fermentation.

Blynk Switch Button ➡ Run Inference

Also, I employed XIAO ESP32C3 to transmit the collected yogurt processing data to the Blynk application every 30 seconds and send the prediction (detection) result after running inferences successfully.

You can download the AI_yogurt_processing_run_model.ino file to try and inspect the code for running Edge Impulse neural network models and communicating with a Blynk application on XIAO ESP32C3.

You can inspect the corresponding functions and settings in Step 4.

⭐ Define the Template ID, Device Name, and Auth Token parameters provided by Blynk.Cloud.

⭐ Include the required libraries.

⭐ Define the required variables for communicating with the Blynk web application and the virtual pins connected to the dashboard widgets.

⭐ Define the required parameters to run an inference with the Edge Impulse model.

⭐ Define the features array (buffer) to classify one frame of data.

⭐ Define the threshold value (0.60) for the model outputs (predictions).

⭐ Define the yogurt consistency level (class) names:

Thinner
Optimum
Curdling

⭐ Define monochrome graphics.

⭐ Create an array including icons for each yogurt consistency level (class).

⭐ Create the Blynk object with the Wi-Fi network settings and the Auth Token parameter.

⭐ Initiate the communication between the Blynk web application (dashboard) and XIAO ESP32C3.

⭐ In the update_Blynk_parameters function, transfer the collected yogurt processing data to the Blynk web application (dashboard).

⭐ Obtain the incoming value from the switch (button) widget on the Blynk dashboard.

⭐ Then, change the model running status depending on the received value (True or False).

⭐ In the run_inference_to_make_predictions function:

⭐ Scale (normalize) the collected data depending on the given model and copy the scaled data items to the features array (buffer).

⭐ If required, multiply the scaled data items while copying them to the features array (buffer).

⭐ Display the progress of copying data to the features buffer on the serial monitor.

⭐ If the features buffer is full, create a signal object from the features buffer (frame).

⭐ Then, run the classifier.

⭐ Print the inference timings on the serial monitor.

⭐ Obtain the prediction (detection) result for each given label and print them on the serial monitor.

⭐ The detection result greater than the given threshold (0.60) represents the most accurate label (yogurt consistency level) predicted by the model.

⭐ Print the detected anomalies on the serial monitor, if any.

⭐ Finally, clear the features buffer (frame).

⭐ If the switch (button) widget on the Blynk dashboard is activated, start running an inference with the Edge Impulse model to predict the yogurt consistency level.

⭐ Then, change the model running status to False.

⭐ If the Edge Impulse model predicts a yogurt consistency level (class) successfully:

⭐ Display the prediction (detection) result (class) on the SSD1306 OLED screen with its assigned monochrome icon.

⭐ Transfer the predicted label (class) to the Blynk web application (dashboard) to inform the user.

⭐ Clear the predicted label.

⭐ Every 30 seconds, transmit the collected environmental factors and culture amount to the Blynk web application so as to update the assigned widgets for each data element on the Blynk dashboard.

Step 7: Running the model on XIAO ESP32C3 to predict yogurt texture levels

My Edge Impulse neural network model predicts possibilities of labels (yogurt consistency classes) for the given features buffer as an array of 3 numbers. They represent the model's "confidence" that the given features buffer corresponds to each of the three different yogurt consistency levels (classes) [0 - 2], as shown in Step 5:

0 — Thinner
1 — Optimum
2 — Curdling

After executing the AI_yogurt_processing_run_model.ino file on XIAO ESP32C3:

🐄🥛📲 The device shows the opening screen if the sensor and MicroSD card module connections with XIAO ESP32C3 are successful.

🐄🥛📲 Then, the device displays the collected environmental factor measurements and the culture (starter) amount on the SSD1306 OLED screen:

Temperature (°C)
Humidity (%)
Pressure (kPa)
Milk Temperature (°C)
Starter Weight (g)

🐄🥛📲 Also, every 30 seconds, the device transmits the collected yogurt processing data to the Blynk web application so as to update the assigned widgets for each data element on the Blynk dashboard.

🐄🥛📲 If the switch (button) widget is activated on the Blynk dashboard, the device runs an inference with the Edge Impulse model and displays the detection result, which represents the most accurate label (yogurt consistency class) predicted by the model.

🐄🥛📲 Each yogurt consistency level (class) has a unique monochrome icon to be shown on the SSD1306 OLED screen when being predicted (detected) by the model:

Thinner
Optimum
Curdling (Lumpy)

🐄🥛📲 After running the inference successfully, the device also transfers the predicted label (class) to the Blynk web application (dashboard) to inform the user.

🐄🥛📲 Also, the device prints notifications and sensor measurements on the serial monitor for debugging.

As far as my experiments go, the device detects yogurt consistency (texture) levels precisely before fermentation :)

After the fermentation process, I had yogurt batches with the exact consistency (texture) levels predicted by the Edge Impulse neural network model.

Videos and Conclusion

Further Discussions

By applying neural network models trained on temperature, humidity, pressure, milk temperature, and culture weight measurements in detecting yogurt consistency (texture) levels, we can achieve to:

🐄🥛📲 improve product quality without food additives,

🐄🥛📲 reduce the total cost for local dairies,

🐄🥛📲 incentivize small businesses to produce organic (natural) yogurt.

References

[^1] Good Food team, Yogurt, BBC Good Food, https://www.bbcgoodfood.com/glossary/yogurt-glossary

[^2] Metabolism Characteristics of Lactic Acid Bacteria and the Expanding Applications in Food Industry, Front. Bioeng. Biotechnol., 12 May 2021, Sec. Synthetic Biology, https://doi.org/10.3389/fbioe.2021.612285

Methane Monitoring in Mines - Silabs xG24 Dev Kit

Detecting the presence of methane using a SiLabs xG24 Dev Kit and a gas sensor.

Created By:

Public Project Link:

GitHub Repository:

Introduction

Methane is a colorless, odorless gas that is the main component of natural gas. It is also a common by-product of coal mining. When methane is present in high concentrations, it can be explosive. For this reason, methane monitoring is essential for the safety of workers in mines and other workplaces where methane may be present.

There are many different ways to monitor methane levels. Some methods, such as fixed gas monitors, are designed to provide constant readings from a set location. Others, such as personal portable gas monitors, are designed to be carried by individual workers so that they can take immediate action if methane levels rise to dangerous levels.

Most countries have regulations in place that require the monitoring of methane levels in mines and other workplaces. These regulations vary from country to country, but they all have the same goal: to keep workers safe from the dangers of methane gas exposure.

The Challenge

There are many different methane monitoring systems on the market, but choosing the right one for your workplace can be a challenge. There are a few things you should keep in mind when choosing a methane monitor:

The type of work environment: Methane monitors come in a variety of shapes and sizes, each designed for a specific type of work environment. You will need to choose a methane monitor that is designed for use in the type of workplace where it will be used. For example, personal portable gas monitors are designed to be worn by individual workers, while fixed gas monitors are designed to be placed in a specific location.
The size of the workplace: The size of the workplace will determine how many methane monitors you will need. For example, a small mine might only require a few fixed gas monitors, while a large mine might require dozens.
The methane concentration: The level of methane present in the workplace will determine how often the methane monitor needs to be used. For example, in a workplace with a high concentration of methane, the monitor may need to be used more frequently than in a workplace with a low concentration of methane.

Choosing the right methane monitor for your workplace can be a challenge, but it is an important part of keeping your workers safe from the dangers of methane gas.

Our Solution

The Silabs EFR32xG24 Dev Kit is the perfect solution for methane monitoring in mines and other workplaces. It features a Machine Learning (ML) hardware accelerator that can be used to develop custom gas detection algorithms. The Silabs EFR32xG24 comes equipped with a pressure sensor, ambient light sensor, and hall-effect sensor, all of which can be used to further customize the monitoring system.

Hardware Requirements

Micro USB cable
3D printed enclosure
Prototyping wires

Software requirements

Edge Impulse account
Edge Impulse CLI

Hardware Setup

To use the MQ-4 sensor with the Silabs EFR32xG24 Dev Kit, you will need to connect the sensor to the board as follows:

Connect the VCC pin on the sensor to the 3.3V pin on the board.
Connect the GND pin on the sensor to the GND pin on the board.
Connect the A0 pin on the sensor to the A0 pin on the board.

With the hardware set up, you are ready to begin developing your methane monitoring system.

Software Setup

Creating the Build Environment

In the next screen, you should see your device name:

Afterwards, select Auto in the Package Installation Options menu and click Next.

This code sample reads analog values from the Methane sensor connected to Pin 16 of the dev board, which corresponds to PC05 (Pin 5, Port C). If you want to change the pin, you’ll have to update the following lines of code:

Next up, right click on the project name in the Project Explorer menu, click on Run as and select Silicon Labs ARM program. The Device Selection menu will pop up and you’ll have to select your board.

In case this warning pops up click Yes:

To check out the values printed by the dev board, you can use the Arduino IDE serial monitor or Picocom.

Creating an Edge Impulse Project

To get started, you will need to create an Edge Impulse project. Edge Impulse is a Machine Learning platform that makes it easy to develop custom algorithms for a variety of applications, including methane monitoring.

To create an Edge Impulse project, simply log in or sign up for an account at https://www.edgeimpulse.com/. Once you have an account, click the "Create new project" button on the dashboard.

You will be asked to give your project a name and select a category. For this project, we will be using the "Custom classification" template. Give your project a name and description, then click the "Create project" button.

With your Edge Impulse project created, you are ready to begin developing your methane detection algorithm.

Connecting the Device

To populate the data pool, connect the board to the computer using a micro-USB cable, launch a terminal and run:

After entering your account information (username and password), you must first choose a project for the device to be assigned to.

You will then be prompted to name the sensor axis that the edge-impulse-data-forwarder picked up.

If everything went well, the development board will appear on your project's Devices tab with a green dot next to it, signifying that it is online and prepared for data collection.

Collecting Training Data

The first step in developing a Machine Learning algorithm is to collect training data. This data will be used to train the algorithm so that it can learn to recognize the patterns that indicate the presence of methane gas.

To collect training data, you will need to use the methane sensor to take readings in a variety of conditions, both with and without methane gas present. For each reading, you will need to take note of the concentration of methane present, as well as the ambient temperature and humidity.

It is important to collect a variety of data points, as this will give the algorithm a better chance of learning to recognize the patterns that indicate the presence of methane gas. Try to take readings in different places, at different times of day, and in different weather conditions. If possible, it is also helpful to take readings with different people so that the algorithm can learn to recognize the patterns that are specific to each situation.

Now, you will need to configure the sensor settings. For this project, we will be using the following settings:

Sensor: MQ-4
Board sampling rate: 1924 Hz
Data recording length: 10 seconds

With these settings configured, click the "Start sampling" button to begin collecting data. The data will be automatically stored in the Edge Impulse cloud and can be used to train your Machine Learning algorithm.

Machine learning algorithms need to be trained on data that is representative of real-world data that they will encounter when deployed on the edge. For this reason, it is important to split the data into training and testing sets, the first of them being used during the training process of the neural network and the other one will be used to evaluate the performance of said NN.

Designing the Impulse

With the training data collected, you are ready to begin designating your methane detection algorithm. In Edge Impulse, algorithms are designed using a drag-and-drop interface called the "Impulse designer".

To access the impulse designer, click the "Design impulse" button on your project's dashboard. You will be presented with a list of available blocks that can be used to build your algorithm. For this project, we will be using the following blocks:

Configuring the Spectral Analysis Block

After you click “Save impulse,” you will notice that each block may be configured by clicking on its name under the “Impulse Design” submenu. The Spectral Analysis block is one of the simplest processing blocks since it only has a few adjustable parameters. On the upper part of the screen, you can see a time-domain representation of the sample that was selected.

Configure the Classification (Keras) Block

So, how does a neural network know what predictions to make? The answer lies in its many layers, where each layer is connected to another through neurons. At the beginning of the training process, connection weights between neurons are randomly determined.

A neural network is designed to predict a set of results from a given set of data, which we call training data. This works by first presenting the network with the training data, and then checking its output against the correct answer. Based on how accurate the prediction was, the connection weights between neurons are adjusted. We repeat this process multiple times until predictions for new inputs become more and more accurate.

When configuring this block, there are multiple parameters that can be modified:

The number of training cycles is defined as the total number of epochs completed in a certain amount of time. Each time the training algorithm makes one complete pass through all of the learning data with back-propagation and modifies the model's parameters as it goes, it is known as an epoch or training cycle (Figure 1).

The learning rate controls how much the model's internal parameters are updated during each step of the training process, or in other words, how quickly the neural network will learn. If the network overfits too rapidly, you can lower the learning rate.

Auto-balance dataset mixes in more copies of data from classes that are uncommon. This function might help make the model more robust against overfitting if you have little data for some classes.

Conclusion

Even though methane leaks in mining operations are a well-known problem with a variety of market-available solutions, conventional threshold-based detection systems have some drawbacks, including the need for remote data processing and the challenges of ensuring connectivity in mining-specific environments.

Edge AI technologies allow users to circumvent those weak points by embedding the whole signal acquisition, data processing and decision making on a single board that can run independently from any wireless network and which can raise an alarm if dangerous trends in the methane levels in the facility are detected.

AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3

Use a XIAO ESP32C3 to monitor temperature, humidity, and pressure to help aid in dairy manufacturing processes.

Created By: Kutluhan Aktar

Public Project Link:

Description

Temperature
Humidity
Pressure
Milk Temperature
Yogurt Culture (Starter) Amount (Weight)

Thinner
Optimum
Curdling (Lumpy)

Lastly, to make the device as sturdy and robust as possible while operating in a dairy, I designed a dairy-themed case with a sliding (removable) front cover (3D printable).

So, this is my project in a nutshell 😃

🎁🎨 Huge thanks to for sponsoring these products:

⭐ XIAO ESP32C3 |

⭐ XIAO Expansion Board |

⭐ Grove - Temperature & Humidity Sensor |

⭐ Grove - Integrated Pressure Sensor Kit |

🎁🎨 Huge thanks to for sponsoring a .

🎁🎨 Also, huge thanks to for sending me a , a , and a .

Step 1: Designing and printing a dairy-themed case

I designed the main case and its sliding front cover in Autodesk Fusion 360. You can download their STL files below.

For the cow figure (replica) affixed to the top of the cylinder part of the main case, I utilized this model from Thingiverse:

Then, I sliced all 3D models (STL files) in Ultimaker Cura.

Since I wanted to create a solid structure for the main case with the sliding front cover representing dairy products, I utilized these PLA filaments:

Beige
ePLA-Matte Milky White

Before the first use, remove unnecessary cable ties and apply grease to the rails.

Test the nozzle and hot bed temperatures.

Go to Print Setup ➡ Auto leveling and adjust five predefined points automatically with the assisted leveling function.

Finally, place the filament into the integrated spool holder and feed the extruder with the filament.

Since the Sermoon V1 is not officially supported by Cura, download the latest version and copy the official printer settings provided by Creality, including Start G-code and End G-code, to a custom printer profile on Cura.

Step 1.1: Improving print quality and speed with the Creality Sonic Pad

Since I wanted to improve my print quality and speed with Klipper, I decided to upgrade my Creality CR-200B 3D Printer with the .

Although the Sonic Pad is pre-configured for some Creality printers, it does not support the CR-200B officially yet. Therefore, I needed to add the CR-200B as a user-defined printer to the Sonic Pad. Since the Sonic Pad needs unsupported printers to be flashed with the self-compiled Klipper firmware before connection, I flashed my CR-200B with the required Klipper firmware settings via FluiddPI by following .

If you do not know how to write a printer configuration file for Klipper, you can download the stock CR-200B configuration file from .

After flashing the CR-200B with the Klipper firmware, copy the configuration file (printer.cfg) to a USB drive and connect the drive to the Sonic Pad.

After setting up the Sonic Pad, select Other models. Then, load the printer.cfg file.

After connecting the Sonic Pad to the CR-200B successfully via a USB cable, the Sonic Pad starts the self-testing procedure, which allows the user to test printer functions and level the bed.

After completing setting up the printer, the Sonic Pad lets the user control all functions provided by the Klipper firmware.

In Cura, export the sliced model in the ufp format. After uploading .ufp files to the Sonic Pad via the USB drive, it converts them to sliced G-code files automatically.

Also, the Sonic Pad can display model preview pictures generated by Cura with the Create Thumbnail script.

Step 1.2: Assembling the case and making connections & adjustments

First of all, I attached XIAO ESP32C3 to . Then, I connected and to the expansion board via Grove connection cables.

Since does not include a compatible connection cable for a Grove port, I connected the weight sensor to the expansion board via a 4-pin male jumper to Grove 4-pin conversion cable.

After printing all parts (models), I fastened all components except the expansion board to their corresponding slots on the main case via a hot glue gun.

I attached the expansion board to the main case by utilizing M3 screws with hex nuts and placed the rubber tube of the integrated pressure sensor in the hollow cylinder part of the main case.

Then, I placed the sliding front cover via the dents on the main case.

Finally, I affixed the small cow figure to the top of the cylinder part of the main case via the hot glue gun.

Step 2: Creating a Blynk application and user interface for XIAO ESP32C3

provides a free cloud service to communicate with supported microcontrollers and development boards, such as ESP32C3. Also, Blynk lets the user design unique web and mobile applications with drag-and-drop editors.

First of all, create an account on and open Blynk.Console.

Before designing the web application on Blynk.Console, install on the Arduino IDE to send and receive data packets via the Blynk cloud: Go to Sketch ➡ Include Library ➡ Manage Libraries… and search for Blynk.

Then, create a new device with the Quickstart Template, named XIAO ESP32C3. And, select the board type as ESP32.

After creating the device successfully, copy the Template ID, Device Name, and Auth Token variables required by the Blynk library.

Open the Web Dashboard and click the Edit button to change the web application design.

From the Widget Box, add the required widgets and assign each widget to a virtual pin as the datastream option.

Since Blynk allows the user to adjust the unit, data range, and color scheme for each widget, I was able to create a unique web user interface for the device.

Temperature Gauge ➡ V4
Humidity Gauge ➡ V12
Pressure Gauge ➡ V6
Milk Temperature Gauge ➡ V7
Weight Gauge ➡ V8
Switch Button ➡ V9
Label ➡ V10

After completing designing the web user interface, I tested the virtual pin connection of each widget with XIAO ESP32C3.

Step 3: Setting up XIAO ESP32C3 on the Arduino IDE

However, before proceeding with the following steps, I needed to set up on the Arduino IDE and install the required libraries for this project.

To add the XIAO ESP32C3 board package to the Arduino IDE, navigate to File ➡ Preferences and paste the URL below under Additional Boards Manager URLs.

https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

Then, to install the required core, navigate to Tools ➡ Board ➡ Boards Manager and search for esp32.

After installing the core, navigate to Tools ➡ Board ➡ ESP32 Arduino and select XIAO_ESP32C3.

Therefore, I needed to change the assigned SS pin to 4 (GPIO4) in the pins_arduino.h file.

The pins_arduino.h file location: \esp32\hardware\esp32\2.0.5\variants\XIAO_ESP32C3.

Finally, download the required libraries for the temperature & humidity sensor, the I2C weight sensor, the DS18B20 temperature sensor, and the SSD1306 OLED display:

Sensirion arduino-core |

arduino-i2c-sht4x |

DFRobot_HX711_I2C |

OneWire |

DallasTemperature |

Adafruit_SSD1306 |

Adafruit-GFX-Library |

Step 3.1: Displaying images on the SSD1306 OLED screen

To display images (black and white) on the SSD1306 OLED screen successfully, I needed to create monochromatic bitmaps from PNG or JPG files and convert those bitmaps to data arrays.

First of all, download the .

Then, upload a monochromatic bitmap and select Vertical or Horizontal depending on the screen type.

Convert the image (bitmap) and save the output (data array).

Finally, add the data array to the code and print it on the screen.

Step 4: Logging yogurt processing information in a CSV file on the SD card w/ XIAO ESP32C3

Temperature (°C)
Humidity (%)
Pressure (kPa)
Milk Temperature (°C)
Starter Weight (g)

Button (short-pressed) ➡ Select a class (Thinner, Optimum, Curdling)
Button (long-pressed) ➡ Save data to the SD card