Links
Comment on page

AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar

A machine learning project to help identify cracks and defects in pipes with the use of mmWave radar and an Arduino Nicla Vision.
Created By: Kutluhan Aktar

Description

Since the beginning of the industrial revolution, accurate pipeline system maintenance has been crucial to keeping machine operations sustainable, profitable, and stable. Even though all machine parts and control units evolved from occupying rooms to fitting in our packets, pipeline system maintenance is still one of the most important aspects of keeping machines healthy while running automated manufacturing operations. From cooling processors with water on motherboards to supplying liquefied metal alloy or plastic for injection molding processes, a faulty pipeline system can engender various manufacturing problems while running machine operations, especially for small businesses with limited budgets not enough to cover expensive overhauling costs.
Therefore, establishing an efficient and accurate pipeline diagnostics mechanism conforming to general maintenance regulations can assist technicians in keeping machines durable far more than anticipated and prevent companies from squandering their resources on replacing or repairing high-value machine components due to the omission of proper pipeline diagnostics.
Pipe cracks are one of the most common defects while transferring liquids, especially with differing thermal conditions. During machine operations, mechanical and thermal stress cause minute defects in pipelines due to fatigue. When these small defects accumulate, the outcome mostly results in a varying inside turbulent pressure, which leads to slight form (shape) disfigurations, resulting in gradual deficiency over time due to tension. Furthermore, depending on operation processes and environment, there are lots of possible pipeline defects in addition to cracks, such as corrosion, abrasion, clogged joints due to chemical residue, leaking connection points due to high gas emissions, etc.
Although there are different external pipeline inspection devices utilizing computer vision (camera), magnetic field measurements, and acoustic detection (microphone)[^1], these methods cannot be applied interchangeably to different pipeline systems. For instance, a device utilizing object detection with a thermal camera may not be able to detect internal crystals due to high gas permeability in a pipeline system transporting antifreeze to cool components.
Nonetheless, some groundbreaking new methods aim to detect potential pipeline system failures by examining changes in the vibration characteristics. Since accumulating stress due to pipeline defects affects material integrity and structure gradually, these failures can be detected by inspecting fluctuating vibrations as a non-destructive testing and evaluation (NDT&E) mechanism. For example, in recent examinations, researchers applied ground penetrating radar (GPR) to detect cracks in a buried pipe[^2] and microwave-based synthetic aperture radar (SAR) to inspect pipeline defects[^3].
After perusing recent research papers on pipeline diagnostics based on vibrations, I noticed there are nearly no appliances focusing on collecting data from a mmWave radar module to extract data parameters, detecting potential pipeline defects, and providing real-time detection results with captured images of the deformed pipes for further examination. Therefore, I decided to build a budget-friendly and compact mechanism to diagnose pipeline defects with machine learning and inform the user of the model detection results with captured images of the deformed pipes simultaneously, in the hope of assisting businesses in keeping machines durable and stable by eliminating basic pipeline defects.
To diagnose different pipeline defects, I needed to collect accurate vibration measurements from a pipeline system so as to train my neural network model with notable validity. Therefore, I decided to build a simple pipeline system by utilizing pipes and fittings (adapters) with mediocre thermal conductivity, demonstrating three different pipeline defects in each primary section — color-coded. Since Seeed Studio provides mmWave radar modules with built-in algorithms to detect minute vibration changes to evaluate respiratory rate, heart rate, and sleep status, I decided to utilize a 60GHz mmWave module to extract my data parameters via the mentioned algorithms. Since Arduino Nicla Vision is a ready-to-use and compact edge device with a 2MP color camera and integrated WiFi/BLE connectivity, I decided to use Nicla Vision so as to run my neural network model, capture images of the deformed pipes, and inform the user of the model detection results with the captured pipe images. Due to architecture and library incompatibilities, I connected the mmWave module to Arduino Nano in order to extract and transmit radar data parameters to Nicla Vision via serial communication. Then, I connected four control buttons to Arduino Nano to send commands with the collected mmWave data parameters to Nicla Vision. Also, I added an ILI9341 TFT LCD screen to display the interface menu, including a custom radar indicator.
Since I focused on building a full-fledged AIoT device diagnosing pipeline system defects, I decided to develop a web application from scratch providing various features to the user. Firstly, I employed the web application to obtain the collected mmWave data parameters with the selected label from Nicla Vision via an HTTP GET request, save the received information to a MySQL database table, and display the stored data records on its interface in descending order. Via a single HTML button on the interface, the web application can also generate a pre-formatted CSV file from the stored data records in the database without requiring any additional procedures.
After completing my data set by collecting data from the custom pipeline system I assembled, I built my artificial neural network model (ANN) with Edge Impulse to make predictions on pipeline system defects (classes). Since Edge Impulse is nearly compatible with all microcontrollers and development boards, I had not encountered any issues while uploading and running my model on Nicla Vision. As labels, I utilized the three basic pipeline defects manifested by each main line (color-coded on the system):
  • Clogged
  • Cracked
  • Leakage
After training and testing my neural network model, I deployed and uploaded the model on Nicla Vision as an Arduino library. Therefore, the device is capable of diagnosing pipeline system defects by running the model independently without any additional procedures or latency.
Then, I utilized the web application to obtain the model detection results with captured images of the deformed pipes from Nicla Vision via HTTP POST requests, save the received information to a particular MySQL database table, and display the stored model results with the assigned detection images on the application interface in descending order simultaneously.
Due to the fact that Nicla Vision can only generate raw image buffer (RGB565), this complementing web application executes a Python script to convert the received raw image buffer to a JPG file automatically before saving it to the server. After saving the converted image successfully, the web application adds it to the HTML table on the interface consecutively, allowing the user to inspect all previous model detection results and the assigned deformed pipe images in descending order.
Considering harsh operating conditions, I decided to design a unique PCB after completing the wiring on a breadboard for the prototype and testing my code and neural network model. Since I wanted my PCB design to emanate a unique and powerful water-damage sensation, I decided to design a Dragonite-inspired PCB since it was the first scary water-related Pokémon for me from the anime, despite being a Dragon/Flying type Pokémon. Thanks to the unique orange solder mask and blue silkscreen combination, only provided by PCBWay, this PCB turned out to be my coolest design yet :)
Since I decided to host my web application on LattePanda 3 Delta, I wanted to build a mobile and compact apparatus to display the web application in the field without requiring an additional procedure. To improve the user experience, I utilized a high-quality 8.8" IPS monitor from Elecrow. As explained in the following steps, I designed a two-part case (3D printable) in which I placed the Elecrow IPS monitor.
Lastly, to make the device as sturdy and compact as possible, I designed an emphasizing liquid-themed case with a sliding front cover and a modular camera holder providing a circular snap-fit joint (3D printable) for Nicla Vision and the 60GHz mmWave radar module.
So, this is my project in a nutshell 😃
In the following steps, you can find more detailed information on coding, capturing deformed pipe images, building a neural network model with Edge Impulse, running the model on Nicla Vision, and developing a full-fledged web application to obtain data parameters with captured images from Nicla Vision via HTTP POST requests.
🎁🎨 Huge thanks to PCBWay for sponsoring this project.
🎁🎨 Huge thanks to Elecrow for sending me an Elecrow 8.8" IPS Monitor (1920*480).
🎁🎨 Huge thanks to DFRobot for sending me a LattePanda 3 Delta 864
🎁🎨 Also, huge thanks to Anycubic for sponsoring a brand-new Anycubic Kobra 2.
image
image
image
image
image
image
image
image
image
image
image
image
image
image
image

Step 1: Designing and soldering the Dragonite-inspired PCB

Before prototyping my Dragonite-inspired PCB design, I tested all connections and wiring with Nicla Vision and Arduino Nano. Then, I checked the data transfer processes between Nicla Vision and the web application hosted on LattePanda 3 Delta.
image
image
Then, I designed my Dragonite-inspired PCB by utilizing KiCad. As mentioned earlier, I wanted to design my PCB based on Dragonite since it was the first Pokémon that made me afraid of a water attack out of nowhere, despite being a Dragon/Flying type Pokémon :) Thanks to the unique orange solder mask and blue silkscreen combination, only provided by PCBWay, this PCB conveys a unique and effective water-damage sensation. I attached the Gerber file of the PCB below: You can order my design from PCBWay to build this device diagnosing pipeline system defects.
image
image
image
First of all, by utilizing a TS100 soldering iron, I attached headers (female), pushbuttons (6x6), resistors (220Ω), a 5mm common anode RGB LED, and a power jack to the PCB.
📌 Component list on the PCB:
A1 (Headers for Arduino Nano)
Nicla1 (Headers for Nicla Vision)
mmWave1 (Headers for 60GHz mmWave Module)
S1 (Headers for ILI9341 TFT LCD Screen)
L1, L2, L3 (Headers for Bi-Directional Logic Level Converter)
K1, K2, K3, K4 (6x6 Pushbutton)
D1 (5mm Common Anode RGB LED)
R1, R2, R3 (220Ω Resistor)
J1 (Power Jack)
image
image
image

Step 1.1: Making connections and adjustments

// Connections
// Arduino Nicla Vision :
// Arduino Nano
// UART_TX (PA_9) --------------- A0
// UART_RX (PA_10) --------------- A1
&
&
&
// Arduino Nano :
// Arduino Nicla Vision
// A0 --------------------------- UART_TX (PA_9)
// A1 --------------------------- UART_RX (PA_10)
// Seeed Studio 60GHz mmWave Sensor
// A2 --------------------------- TX
// A3 --------------------------- RX
// 2.8'' 240x320 TFT LCD Touch Screen (ILI9341)
// D10 --------------------------- CS
// D9 --------------------------- RESET
// D8 --------------------------- D/C
// D11 --------------------------- SDI (MOSI)
// D13 --------------------------- SCK
// 3.3V --------------------------- LED
// D12 --------------------------- SDO(MISO)
// Control Button (A)
// D2 --------------------------- +
// Control Button (B)
// D4 --------------------------- +
// Control Button (C)
// D7 --------------------------- +
// Control Button (D)
// A4 --------------------------- +
// 5mm Common Anode RGB LED
// D3 --------------------------- R
// D5 --------------------------- G
// D6 --------------------------- B
After completing soldering, I attached all remaining components to the Dragonite PCB via headers — Nicla Vision, Arduino Nano, 60GHz mmWave radar module, bi-directional logic level converters, and ILI9341 TFT LCD screen.
Due to architecture and library incompatibilities, I decided to connect the mmWave module and the ILI9341 TFT LCD screen to Arduino Nano so as to extract and display data parameters. Then, I utilized Arduino Nano to transmit the collected mmWave data parameters and user commands to Nicla Vision via serial communication.
Since Arduino Nano operates at 5V and Nicla Vision requires 3.3V logic level voltage, they cannot be connected with each other directly. Therefore, I utilized a bi-directional logic level converter to shift the voltage for the connections between Nicla Vision and Arduino Nano.
Even though the ILI9341 TFT screen can be supplied with 5V, it generates 3.3V interface voltage. Therefore, connecting its SPI pins to a 5V board like Arduino Nano leads to black screen and freezing issues. To make the ILI9341 screen stable, I utilized two bi-directional logic level converters shifting the voltage for the connections between the ILI9341 screen and Arduino Nano.
To be able to transfer commands to Nicla Vision via serial communication, I connected the software serial port of Arduino Nano (Nicla) to the hardware serial port of Nicla Vision (Serial1). To communicate with the MR60BHA1 60GHz radar module, I connected the software serial port of Arduino Nano (mmWave) to the module's built-in UART interface.
I utilized the ILI9341 TFT screen to display ongoing operations and visualize the extracted mmWave data parameters by creating a simple radar indicator. Then, I added four control buttons to transfer the collected data parameters and user commands to Nicla Vision via serial communication. Also, I added an RGB LED to inform the user of the device status, denoting serial communication and data collection success.
image
image
image

Step 2: Designing and printing a liquid-themed case w/ Anycubic Kobra 2

Since I focused on building a user-friendly and accessible mechanism that collects mmWave data parameters and runs a neural network model to inform the user of the diagnosed pipeline defects via a PHP web application, I decided to design a rigid and compact case allowing the user to place the 60GHz mmWave module and position the built-in GC2145 camera on Nicla Vision effortlessly. To avoid overexposure to dust and prevent loose wire connections, I added a sliding front cover aligned proportionally to the diagonal top surface. Then, I designed a modular camera holder mountable to the back of the case via a circular snap-fit joint. Also, I decided to emboss pipe icons and the Arduino symbol on the sliding front cover to emphasize the edge pipeline diagnostic processes.
Since I needed to attach the Dragonite PCB to the main case, I decided to design an oblique structure for the case. In that regard, I was able to fit the PCB in the case without enlarging the case dimensions.
I designed the main case, its sliding front cover, and the modular camera holder in Autodesk Fusion 360. You can download their STL files below.
image
image
image
image
image
image
image
image
image
image
image
Then, I sliced all 3D models (STL files) in Ultimaker Cura.
image
image
image
image
image
image
Since I wanted to create a shiny structure for the main case and apply a unique liquid theme indicating water damage, I utilized these PLA filaments:
  • eSilk Cyan
  • ePLA-Matte Light Blue
Finally, I printed all parts (models) with my brand-new Anycubic Kobra 2 3D Printer.
image
image
Since Anycubic Kobra 2 is budget-friendly and specifically designed for high-speed printing, I highly recommend Anycubic Kobra 2 if you are a maker or hobbyist needing to print multiple prototypes before finalizing a complex project.
Thanks to its upgraded direct extruder, Anycubic Kobra 2 provides 150mm/s recommended print speed (up to 250mm/s) and dual-gear filament feeding. Also, it provides a cooling fan with an optimized dissipation design to support rapid cooling complementing the fast printing experience. Since the Z-axis has a double-threaded rod structure, it flattens the building platform and reduces the printing layers, even at a higher speed.
Furthermore, Anycubic Kobra 2 provides a magnetic suction platform on the heated bed for the scratch-resistant spring steel build plate allowing the user to remove prints without any struggle. Most importantly, you can level the bed automatically via its user-friendly LeviQ 2.0 automatic bed leveling system. Also, it has a smart filament runout sensor and the resume printing function for power failures.
#⃣
First of all, install the gantry and the spring steel build plate.
image
image
#⃣
Install the print head, the touch screen, and the filament runout sensor.
image
image
#⃣
Connect the stepper, switch, screen, and print head cables. Then, attach the filament tube.
image
#⃣
If the print head is shaking, adjust the hexagonal isolation column under the print head.
image
#⃣
Go to Prepare➡ Leveling ➡ Auto-leveling to initiate the LeviQ 2.0 automatic bed leveling system.
#⃣
After preheating and wiping the nozzle, Anycubic Kobra 2 probes the predefined points to level the bed.
image
image
image
image
image
#⃣
Finally, fix the filament tube with the cable clips, install the filament holder, and insert the filament into the extruder.
image
#⃣
Since Anycubic Kobra 2 is not officially supported by Cura yet, download the latest PrusaSlicer version and import the printer profile (configuration) file provided by Anycubic.
#⃣
Then, create a custom printer profile on Cura for Anycubic Kobra 2 and change Start G-code and End G-code.
#⃣
Based on the provided Start G-code and End G-code in the configuration file, I modified new Start G-code and End G-code compatible with Cura.
Start G-code:
G90 ; use absolute coordinates
M83 ; extruder relative mode
G28 ; move X/Y/Z to min endstops
G1 Z2.0 F3000 ; lift nozzle a bit
G92 E0 ; Reset Extruder
G1 X10.1 Y20 Z0.28 F5000.0 ; Move to start position
G1 X10.1 Y200.0 Z0.28 F1500.0 E15 ; Draw the first line
G1 X10.4 Y200.0 Z0.28 F5000.0 ; Move to side a little
G1 X10.4 Y20 Z0.28 F1500.0 E30 ; Draw the second line
G92 E0 ; zero the extruded length again
G1 E-2 F500 ; Retract a little
M117
G21 ; set units to millimeters
G90 ; use absolute coordinates
M82 ; use absolute distances for extrusion
G92 E0
M107
End G-code:
M104 S0 ; Extruder off
M140 S0 ; Heatbed off
M107 ; Fan off
G91 ; relative positioning
G1 E-5 F3000
G1 Z+0.3 F3000 ; lift print head
G28 X0 F3000
M84 ; disable stepper motors
#⃣
Finally, adjust the official printer settings depending on the filament type while copying them from PrusaSlicer to Cura.
image
image
image
image
image
image
image

Step 2.1: Assembling the liquid-themed case

After printing all parts (models), I fastened Dragonite PCB to the diagonal top surface of the main case via a hot glue gun.
I placed Nicla Vision and the 60GHz mmWave module in the modular camera holder. Then, I attached the camera holder to the main case via its circular snap-fit joint.
Finally, I inserted the sliding front cover via the dents on the diagonal top surface of the main case.
image
image
image
image
image
image
image
image
image
image
As mentioned earlier, the modular camera holder can be utilized to place the 60GHz mmWave module and position the built-in GC2145 camera on Nicla Vision.
image
image
image

Step 2.2: Creating a LattePanda Deck to display the web application

Since I decided to utilize the web application to display the collected data parameters, generate the pre-formatted CSV file from the stored data records in the database, and show the model detection results with the captured images of the deformed pipes, I wanted to create a unique apparatus to inspect the web application.
Since I host this web application on my LattePanda 3 Delta, I decided to design a unique and compact LattePanda Deck compatible with not only LattePanda but also any single-board computer supporting HDMI.
I decided to employ Elecrow's 8.8" (1920*480) high-resolution IPS monitor as the screen of my LattePanda Deck. Thanks to its converter board, this monitor can be powered via a USB port and works without installing any drivers. Therefore, it is a compact plug-and-play monitor for LattePanda 3 Delta, providing high resolution and up to 60Hz refresh rate.
Due to the fact that I wanted to build a sturdy and easy-to-use deck, I designed a two-part case covering the screen frame and providing a slot for the converter board. To avoid overexposure to dust and provide room for cable management, I added a mountable back cover adorned with the brand logo.
I designed the two-part case and its mountable back cover in Autodesk Fusion 360. You can download their STL files below.
image
image
image
image
image
image
image
Then, I sliced all 3D models (STL files) in Ultimaker Cura.
image
image
image
After printing all deck parts (models) with my Anycubic Kobra 2 3D Printer, I affixed the two-part case together via the hot glue gun.
Then, I fastened the Elecrow's IPS monitor to the case covering the screen frame and inserted the converter board into its slot.
After attaching the required cables to the converter board, I fixed the mountable back cover via M3 screws.
image
image
image
image
image
image
image
image
After connecting the converter board to LattePanda 3 Delta via its USB and HDMI ports, LattePanda recognizes the IPS monitor automatically.
image

Step 3: Developing a web application displaying real-time database updates in PHP, JavaScript, CSS, and MySQL

To provide an outstanding user interface for generating data samples and displaying model detection results, I developed a full-fledged web application from scratch in PHP, HTML, JavaScript, CSS, and MySQL.
First of all, the web application obtains the extracted mmWave data parameters and the selected pipeline diagnostic class from Nicla Vision via an HTTP GET request. After storing the received information in a particular MySQL database table, the web application lets the user generate a CSV file, including all stored data records as samples.
Also, the web application gets the inference data parameters, the diagnosed pipeline defect (class) by the neural network model, and the captured image of the deformed pipe from Nicla Vision via an HTTP POST request. After saving the received information to a particular MySQL database table for further inspection, the web application converts the received raw image buffer (RGB565) to a JPG file by executing a Python script. Then, the web application updates its interface automatically to show the latest stored information in database tables. On the interface, the application shows model detection results, the assigned detection images, and the collected data parameters in descending order so as to allow the user to check previous records easily.
As shown below, the web application consists of three folders and seven code files:
  • /assets
  • -- background.jpg
  • -- class.php
  • -- data_records.csv
  • -- icon.png
  • -- index.css
  • -- index.js
  • /detections
  • -- /images
  • -- rgb565_converter.py
  • index.php
  • show_records.php
  • update_server.php
image
image
image
📁 class.php
In the class.php file, I created a class named _main to bundle the following functions under a specific structure.
⭐ Define the _main class and its functions.
class _main {
public $conn;
public function __init__($conn){
$this->conn = $conn;
}
⭐ In the insert_new_data function, insert the given data parameters extracted from a 60GHz mmWave sensor and the selected class to the entries MySQL database table.
public function insert_new_data($date, $mmWave, $class){
$sql_insert = "INSERT INTO `entries`(`date`, `mmwave`, `class`)
VALUES ('$date', '$mmWave', '$class');"
;
if(mysqli_query($this->conn, $sql_insert)){ return true; } else{ return false; }
}
⭐ In the insert_new_results function, insert the given model detection results and assigned detection image name to the detections MySQL database table.
public function insert_new_results($date, $mmWave, $img, $class){
$sql_insert = "INSERT INTO `detections`(`date`, `mmwave`, `img`, `class`)
VALUES ('$date', '$mmWave', '$img', '$class');"
;
if(mysqli_query($this->conn, $sql_insert)){ return true; } else{ return false; }
}
⭐ In the get_data_records function, retrieve all data records from the entries database table in descending order and return each column as separate lists.
public function get_data_records(){
$date=[]; $mmWave=[]; $class=[];
$sql_data = "SELECT * FROM `entries` ORDER BY `id` DESC";
$result = mysqli_query($this->conn, $sql_data);
$check = mysqli_num_rows($result);
if($check > 0){
while($row = mysqli_fetch_assoc($result)){
array_push($date, $row["date"]);
array_push($mmWave, $row["mmwave"]);
array_push($class, $row["class"]);
}
return array($date, $mmWave, $class);
}else{
return array(["Not Found!"], ["Not Found!"], ["Not Found!"]);
}
}
⭐ In the get_model_results function, retrieve all model detection results and the assigned detection image names from the detections database table in descending order. Then, return each column as separate lists.
public function get_model_results(){
$date=[]; $mmWave=[]; $class=[]; $img=[];
$sql_data = "SELECT * FROM `detections` ORDER BY `id` DESC";
$result = mysqli_query($this->conn, $sql_data);
$check = mysqli_num_rows($result);
if($check > 0){
while($row = mysqli_fetch_assoc($result)){
array_push($date, $row["date"]);
array_push($mmWave, $row["mmwave"]);
array_push($class, $row["class"]);
array_push($img, $row["img"]);
}
return array($date, $mmWave, $class, $img);
}else{
return array(["Not Found!"], ["Not Found!"], ["Not Found!"], ["icon.png"]);
}
}
⭐ In the create_CSV function:
⭐ Get the stored data records in the entries database table.
⭐ Create a CSV file — data_records.csv.
⭐ Add the header to the created CSV file.
⭐ Generate rows from the retrieved data records.
⭐ Append each generated row to the CSV file as a sample.
⭐ Close the CSV file and return its name.
public function create_CSV(){
// Get the stored data records in the entries database table.
$date=[]; $mmWave=[]; $label=[];
list($date, $mmWave, $label) = $this->get_data_records();
// Create the data_records.csv file.
$filename = "assets/data_records.csv";
$fp = fopen($filename, 'w');
// Add the header to the CSV file.
fputcsv($fp, ["p_1","p_2","p_3","p_4","p_5","p_6","p_7","pipe_label"]);
// Generate rows from the retrieved data records.
for($i=0; $i<count($date); $i++){
$line = explode(",", $mmWave[$i]);
array_push($line, $label[$i]);
// Append each generated row to the CSV file as a sample.
fputcsv($fp, $line);
}
// Close the CSV file.
fclose($fp);
// Return the CSV file name — data_records.csv.
return $filename;
}
⭐ Define the required MariaDB database connection settings for LattePanda 3 Delta 864.
$server = array(
"name" => "localhost",
"username" => "root",
"password" => "",
"database" => "pipeline_diagnostics"
);
$conn = mysqli_connect($server["name"], $server["username"], $server["password"], $server["database"]);
📁 update_server.php
⭐ Include the class.php file.
⭐ Define the wave object of the _main class with its required parameters.
include_once "assets/class.php";
// Define the new 'wave' object:
$wave = new _main();
$wave->__init__($conn);
⭐ Get the current date & time and create the detection image file name.
$date = date("Y_m_d_H_i_s");
# Create the image file name.
$img_file = "PIPE_".$date;
⭐ If Nicla Vision sends the collected 60GHz mmWave sensor data parameters with the selected pipeline diagnostic class, save the received information to the entries MySQL database table.
if(isset($_GET["data"]) && isset($_GET["mmWave"]) && isset($_GET["class"])){
if($wave->insert_new_data($date, $_GET["mmWave"], $_GET["class"])){
echo "New Data Record Saved Successfully!";
}else{
echo "Database Error!";
}
}
⭐ If Nicla Vision transmits the model detection results, save the received information to the detections MySQL database table.
if(isset($_GET["results"]) && isset($_GET["mmWave"]) && isset($_GET["class"])){
if($wave->insert_new_results($date, $_GET["mmWave"], $img_file.".jpg", $_GET["class"])){
echo "Detection Results Saved Successfully!";
}else{
echo "Database Error!";
}
}
⭐ If Nicla Vision transfers an image of a deformed pipe via an HTTP POST request to update the server after running the neural network model, save the received raw image buffer (RGB565) as a TXT file to the detections folder.
⭐ Then, convert the recently saved RGB565 buffer (TXT file) to a JPG image file by executing a Python script via the terminal through the web application — rgb565_converter.py.
⭐ After generating the JPG file from the raw image buffer, remove the converted TXT file from the server.
if(!empty($_FILES["captured_image"]['name'])){
// Image File:
$captured_image_properties = array(
"name" => $_FILES["captured_image"]["name"],
"tmp_name" => $_FILES["captured_image"]["tmp_name"],
"size" => $_FILES["captured_image"]["size"],
"extension" => pathinfo($_FILES["captured_image"]["name"], PATHINFO_EXTENSION)
);
// Check whether the uploaded file extension is in the allowed file formats.
$allowed_formats = array('jpg', 'png', 'bmp', 'txt');
if(!in_array($captured_image_properties["extension"], $allowed_formats)){
echo 'FILE => File Format Not Allowed!';
}else{
// Check whether the uploaded file size exceeds the 5 MB data limit.
if($captured_image_properties["size"] > 5000000){
echo "FILE => File size cannot exceed 5MB!";
}else{
// Save the uploaded file (image).
move_uploaded_file($captured_image_properties["tmp_name"], "./detections/".$img_file.".".$captured_image_properties["extension"]);
echo "FILE => Saved Successfully!";
}
}
// Convert the recently saved RGB565 buffer (TXT file) to a JPG image file by executing the rgb565_converter.py file.
$raw_convert = shell_exec('python "C:\Users\kutlu\New E\xampp\htdocs\pipeline_diagnostics_interface\detections\rgb565_converter.py"');
print($raw_convert);
// After generating the JPG file, remove the recently saved TXT file from the server.
unlink("./detections/".$img_file.".txt");
}
⭐ If requested, create a CSV file from the data records saved in the entries database table — data_records.csv. Then, download the generated CSV file automatically.
if(isset($_GET["create_CSV"])){
// Create the data_records.csv file.
$filename = $wave->create_CSV();
// Download the generated CSV file automatically.
header('Content-Description: File Transfer');
header('Content-Type: application/octet-stream');
header("Cache-Control: no-cache, must-revalidate");
header('Content-Disposition: attachment; filename="'.basename($filename).'"');
header('Content-Length: '.filesize($filename));
header('Pragma: public');
readfile($filename);
}
📁 show_records.php
⭐ Include the class.php file.
⭐ Define the wave object of the _main class with its required parameters.
include_once "assets/class.php";
// Define the new 'wave' object:
$wave = new _main();
$wave->__init__($conn);
⭐ Obtain all data records from the entries database table as separate lists for each data parameter (column) and create HTML table rows by utilizing these arrays.
$date=[]; $mmWave=[]; $label=[];
list($date, $mmWave, $label) = $wave->get_data_records();
$records = "<tr><th>Date</th><th>mmWave</th><th>Label</th></tr>";
for($i=0; $i<count($date); $i++){
$records .= '<tr class="'.$label[$i].'">
<td>'.$date[$i].'</td>
<td style="word-break:break-all;">'.$mmWave[$i].'</td>
<td>'.$label[$i].'</td>
</tr>
';
}
⭐ Fetch all model detection results with the assigned detection image names from the detections database table as separate lists for each column and generate HTML table rows by utilizing these arrays.
$date_R=[]; $mmWave_R=[]; $class=[]; $img=[];
list($date_R, $mmWave_R, $class, $img) = $wave->get_model_results();
$results = "<tr><th>Date</th><th>mmWave</th><th>Model Prediction</th><th>IMG</th></tr>";
for($i=0; $i<count($date_R); $i++){
$results .= '<tr class="'.$class[$i].'">
<td>'.$date_R[$i].'</td>
<td style="word-break:break-all;">'.$mmWave_R[$i].'</td>
<td>'.$class[$i].'</td>
<td><img src="detections/images/'.$img[$i].'"/></td>
</tr>
';
}
⭐ Then, create a JSON object with multiple key/value pairs from the generated HTML table rows consisting of the retrieved data records and the fetched model detection results.
$result = array("records" => $records, "results" => $results);
$res = json_encode($result);
⭐ Finally, return the recently created JSON object.
echo($res);
📁 index.php
⭐ Create the web application interface to display the stored information in the MySQL database tables with the captured deformed pipe images as HTML table rows.
You can inspect and download the index.php file below.
📁 index.js (jQuery and AJAX)
⭐ If requested, open a confirmation window to download the generated CSV file (data_records.csv) consisting of the data records saved in the entries database table as samples.
$(".records").on("click", "button", () => {
if(confirm("💻 Download the generated CSV file!\n\n🗂 data_records.csv")){
window.location = "./update_server.php?create_CSV=OK";
}
});
⭐ Every 5 seconds, make an HTTP GET request to the show_records.php file.
⭐ Then, decode the retrieved JSON object to obtain the HTML table rows generated from the MySQL database table rows.
⭐ Assign the obtained table rows to the corresponding HTML elements on the web application interface to inform the user of the recently collected data parameters and the latest model detection results automatically.
setInterval(function(){
$.ajax({
url: "./show_records.php",
type: "GET",
success: (response) => {
// Decode the obtained JSON object.
const res = JSON.parse(response);
// Assign the data record HTML table rows.
$(".records table").html(res.records);
// Assign the model detection HTML table rows.
$(".results table").html(res.results);
}
});
}, 5000);
image
image
image
image
image
image
image
image

Step 3.1: Converting the raw image buffers transferred by Nicla Vision via POST requests to JPG files

Since Nicla Vision can only produce raw image buffers (RGB565) due to its built-in 2-megapixel CMOS camera (GC2145) and camera library, I needed to convert the generated raw image buffers to readable image files so as to display them on the web application interface. Since Nicla Vision cannot convert the generated raw image buffer due to memory allocation issues, I decided to convert the captured raw image buffer to a JPG file through the web application.
Even though PHP can handle converting image buffers to different file formats, converting images in PHP causes bad request issues since the web application receives raw image buffers from Nicla Vision via HTTP POST requests. Hence, I decided to utilize Python to create JPG files from raw image buffers since Python provides built-in modules for image conversion in seconds, even for byte swapping.
By employing the terminal on LattePanda 3 Delta, the web application executes the rgb565_converter.py file directly to convert raw image buffers.
#⃣
Since the numpy module is required to convert uint16_t to 3x8-bit pixels, you may require to install the numpy module on the terminal manually to avoid errors.
python -m pip install numpy
📁 rgb565_converter.py
⭐ Include the required modules.
from glob import glob
import numpy as np
from PIL import Image
⭐ Obtain all RGB565 buffer arrays transferred by Nicla Vision as text (TXT) files under the detections folder.
#⃣
Since the web application requires to access the absolute paths via the terminal to execute the Python script in order to convert images, provide the detections folder's exact location.
path = "C:\\Users\\kutlu\\New E\\xampp\\htdocs\\pipeline_diagnostics_interface\\detections"
images = glob(path + "/*.txt")
⭐ Then, convert each retrieved TXT file (RGB565 buffer array) to a JPG file via the frombytes function.
⭐ Finally, save the generated JPG files to the images folder.
  • RGB565 (uint16_t) ➜ RGB (3x8-bit pixels, true color)
for img in images:
loc = path + "/images/" + img.split("\\")[8].split(".")[0] + ".jpg"
size = (320,320)
# RGB565 (uint16_t) to RGB (3x8-bit pixels, true color)
raw = np.fromfile(img).byteswap(True)
file = Image.frombytes('RGB', size, raw, 'raw', 'BGR;16', 0, 1)
file.save(loc)
#print("Converted: " + loc)
image
image

Step 3.2: Setting and running the web application on LattePanda 3 Delta 864

Since I have got a test sample of the brand-new LattePanda 3 Delta 864, I decided to host my web application on LattePanda 3 Delta. Therefore, I needed to set up a LAMP web server.
LattePanda 3 Delta is a pocket-sized hackable computer that provides ultra performance with the Intel 11th-generation Celeron N5105 processor.
Plausibly, LattePanda 3 Delta can run the XAMPP application. So, it is effortless to create a server with a MariaDB database on LattePanda 3 Delta.
As explained in the previous steps, I also designed a unique deck for LattePanda by utilizing Elecrow's 8.8" IPS monitor.
image
image
#⃣
First of all, install and set up the XAMPP application.
#⃣
Then, go to the XAMPP Control Panel and click the MySQL Admin button.
#⃣
Once the phpMyAdmin tool pops up, create a new database named pipeline_diagnostics.
image
image
#⃣
After adding the database successfully, go to the SQL section to create two different MySQL database tables named entries and detections with the required data fields.
CREATE TABLE `entries`(
id int AUTO_INCREMENT PRIMARY KEY NOT NULL,
`date` varchar(255) NOT NULL,
mmwave varchar(255) NOT NULL,
`class` varchar(255) NOT NULL
);
CREATE TABLE `detections`(
id int AUTO_INCREMENT PRIMARY KEY NOT NULL,
`date` varchar(255) NOT NULL,
mmwave varchar(255) NOT NULL,
img varchar(255) NOT NULL,
`class` varchar(255) NOT NULL
);
image
image
image
#⃣
When Nicla Vision transmits the collected 60GHz mmWave sensor data parameters with the selected pipeline diagnostic class, the web application saves the received information to the entries MySQL database table.
image
#⃣
When Nicla Vision transfers the model detection results and the captured image of the deformed pipe, the web application saves the received information to the detections MySQL database table.
image

Step 3.3: Generating data samples and displaying real-time model detection results transferred by Nicla Vision

After setting the web application on LattePanda 3 Delta 864:
🚿🔎📲 The web application (update_server.php) saves the mmWave data parameters with the selected class transferred by Nicla Vision via an HTTP GET request to the entries MySQL database table.
/pipeline_diagnostics_interface/update_server.php?data=OK&mmWave=32.06314106,65.51403019,27.04366461,0.59400105,0.58607824,5.429632,0.81743312&class=Cracked
image
🚿🔎📲 When Nicla Vision transmits raw image buffer (RGB565), model detection results, and inference data parameters via an HTTP POST request with URL query parameters, the web application (update_server.php) stores the received information in the detections MySQL database table. Then, the application converts the received raw image buffer to a JPG file by executing the rgb565_converter.py file via the terminal.
python "C:\Users\kutlu\New E\xampp\htdocs\pipeline_diagnostics_interface\detections\rgb565_converter.py"
🚿🔎📲 On the web application interface (index.php), the application displays the concurrent list of data records saved in the entries database table as an HTML table.
🚿🔎📲 When the user clicks the HTML button (Create CSV), the application opens a confirmation window to generate and download a CSV file (data_records.csv) consisting of the data records saved in the entries database table as samples.
🚿🔎📲 Also, on the application interface, the application shows the concurrent list of model detection results with the captured images of the deformed pipes to inform the user of the latest diagnosed pipeline defects.
🚿🔎📲 The web application updates its interface every 5 seconds automatically via the jQuery script to display the latest stored information in the MariaDB database on LattePanda 3 Delta.
🚿🔎📲 For each pipeline diagnostic class (label), the web application changes the row color in the HTML tables to clarify and emphasize the collected mmWave data parameters and the model detection results:
  • Leakage ➜ Dark Blue
  • Cracked ➜ Violet
  • Clogged ➜ Red
image
image
image
image
image
image
image
image
image
🚿🔎📲 When the user hovers the cursor over the image frames, the web application highlights the selected frame.
image
🚿🔎📲 If Nicla Vision has not transferred any information yet, the web application notifies the user through the HTML tables.
image

Step 4: Setting up Nicla Vision on Arduino IDE

Before proceeding with the following steps, I needed to set up Nicla Vision on the Arduino IDE and install the required libraries for this project.
Although Arduino provides an official board package and libraries for Nicla Vision, the Wi-Fi firmware is not installed out of the box. Therefore, I had to install the Wi-Fi firmware manually to utilize the built-in Wi-Fi module.
#⃣
To install the required core for Nicla boards, navigate to Tools ➡ Board ➡ Boards Manager and search for Arduino Mbed OS Nicla Boards.
image
image
#⃣
After installing the core, navigate to Tools ➡ Board ➡ Arduino Mbed OS Nicla Boards and select Arduino Nicla Vision.
image
#⃣
Since the Wi-Fi firmware has not been installed yet out of the box, Nicla Vision throws an error while attempting to utilize the built-in Wi-Fi module.
image
#⃣
To install the Wi-Fi firmware manually, go to Examples ➡ STM32H747_System ➡ WiFiFirmwareUpdater and execute the provided code.
image
image
#⃣
After running the WiFiFirmwareUpdater example, Arduino IDE flashes Nicla Vision to install the required Wi-Fi firmware and certificates.
image
image
#⃣
Finally, download the required libraries to utilize the 60GHz mmWave module and the ILI9341 TFT LCD screen with Arduino Nano:
MR60BHA1-Sensor | Download
Adafruit_ILI9341 | Download
Adafruit-GFX-Library | Download