Project Demo
Overview

Cover Image

Bridge Collapsed
Why localization?
Why do we want to localize the detection using an image classification model? Can’t we use the object detection model? Yes, we can use the object detection model but we would need to add bounding boxes to thousands of samples manually. Existing object detection models may not be a good choice to auto-annotate these cracks since they are trained on definite shape objects. Repurposing the classification model for localizing the detection saves a lot of effort and still would be able to identify the regions of interest.How does it work?
The CNN (convolutional neural networks) with GAP (Global Average Pooling) layers that have been trained for a classification task can also be used for object localization. That is, a GAP-CNN not only tells us what object is contained in the image - it also tells us where the object is in the image, and through no additional work on our part! The localization is expressed as a heat map (class activation map) where the color-coding scheme identifies regions that are relatively important for the GAP-CNN to perform the object identification task.Hardware Setup
Since I wanted a compact and portable hardware setup, we will be using Seeed reTerminal which comes with an LCD and buttons in a compact form. It is powered by a Raspberry Pi 4 Compute Module with 4 GB RAM which would be sufficient for this proof-of-concept project. We would need Raspberry Pi Camera V2 and an acrylic mount for it.
Hardware

FPC

with Camera
Setup Development Environment
The reTerminal comes with 32-bit Raspberry Pi OS but We will be using 64-bit Raspberry Pi OS for better performance. Please follow the instructions here to flash the 64-bit Raspberry Pi OS: https://wiki.seeedstudio.com/reTerminal-FAQ. To install the python packages which we will be using in the inferencing code, execute the commands below.Data collection
The datasets were downloaded from the Mendeley Data (Concrete Crack Images for Classification). The dataset contains various concrete surfaces with and without cracks. The data is collected from multiple METU Campus Buildings. The dataset is divided into two negative and positive crack images for image classification. Each class has 20,000 images with a total of 40,000 images with 227 x 227 pixels with RGB channels.
Datasets
- Surface Crack Dataset: https://data.mendeley.com/datasets/5y9wdsg2zt/2
- COCO-Minitrain dataset: https://github.com/giddyyupp/coco-minitrain
Uploading data to Edge Impulse Studio
We need to create a new project to upload data to Edge Impulse Studio.
New Project
- Positive - surface with crack
- Negative - surface without crack
- Unknown - images from the 80 objects

Data Acquisition
Training
Go to the Impulse Design > Create Impulse page, click Add a processing block, and then choose Image, which preprocesses and normalizes image data, and optionally reduces the color depth. Also, on the same page, click Add a learning block, and choose Transfer Learning (Images), which fine-tunes a pre-trained image classification model on the data. We are using a 160x160 image size. Now click on the Save Impulse button.
Create Impulse

Feature Generation

Feature Explorer

Model Selection

Switch Expert Mode

Editor

Network

Confusion Matrix
Model Deployment
Currently, Edge Impulse for Linux SDK does not support a multi-output model so we will be using the compiled TensorFlow Lite runtime for inferencing. This interpreter-only package is a fraction of the size of the complete TensorFlow package and includes the bare minimum code required to run inferences with TensorFlow Lite. To accelerate the inferencing, the TFLite interpreter can be used with XNNPACK which is a highly optimized library of neural network inference operators for ARM, and other platforms. To enable XNNPACK for 64-bit Raspberry Pi OS, we need to build the TFLite Runtime Python package from the source. We will need to execute the following commands on a faster Debian/Ubuntu Linux machine with Docker to cross-compile and build the package.tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
file.

Download Model
Application Workflow Diagram
The application implements multithreading to use all available 4-cores on the Raspberry Pi 4 Compute Module to achieve low latency and better FPS.
Workflow Diagram
The Desktop App
The inferencing script is executed by clicking on the Desktop App icon which is created by adding anei.desktop
file at the /home/pi/Desktop
directory.

Desktop App
- F1 button: to toggle heat map
- F2 button: to toggle center crop (zoom in) the preview image
- O button: to close the app
Conclusion
This project showcases an industrial use case for surface crack detection which can be used for predictive maintenance. The project has the following key characteristics.- Customize the pre-trained transfer learning model in the Edge Impulse Studio Expert Mode
- Demonstrate use of a multi-output model trained using Edge Impulse
- Runtime heat-map visualization to localize the detected cracks.
- Multi-threaded application to increase FPS
- A scalable and portable solution