Multi-camera Video Stream Inference - Brainchip Akida
Real-time inferencing for multi-camera video streaming using Brainchip Akida and Edge Impulse FOMO.
Last updated
Real-time inferencing for multi-camera video streaming using Brainchip Akida and Edge Impulse FOMO.
Last updated
Created By: Naveen Kumar
Public Project Link: https://studio.edgeimpulse.com/public/298672/latest
Real-time inferencing for multi-camera video streaming for road crossings is a challenging task that involves processing and analyzing multiple video sources simultaneously, such as from different cameras or sensors, to provide useful information and insights for traffic management, safety, and planning. Some of the possible goals and applications of this task could be:
To recognize and monitor traffic events, such as congestion, accidents, violations, or anomalies, and alert the authorities or the public in real time.
To measure and optimize traffic flow, density, and patterns, and provide guidance or recommendations for traffic control, routing, or scheduling.
In this project, we aim to build a pipeline that is both flexible and scalable for streaming and inference of multi-camera videos. We will utilize BrainChip’s Akida Development Kit. to optimize and accelerate the Edge Impulse FOMO model near the data sources to decrease the bandwidth and latency requirements for video streaming and analytics. The BrainChip's neuromorphic processor IP uses event-based technology for increased energy efficiency. It allows incremental learning and high-speed inference for various applications, including convolutional neural networks, with exceptional performance and low power consumption. The kit includes a PCIe board with an onboard AKD1000 Akida processor, a Raspberry Pi Compute Module 4 (Wi-Fi, 8 GB RAM), and a Raspberry Pi Compute Module 4 I/O (carrier) board. The kit is shown below in a disassembled state.
The Akida PCIe board is connected to the Raspberry Pi Compute Module 4 IO Board through the PCIe Gen 2 x1 socket available onboard.
The Brainchip Akida Kit comes with Ubuntu 20.04 LTS and Akida PCIe drivers preinstalled but we wanted to try the latest Raspberry Pi OS (Bookworm). We can use the Raspberry Pi Imager to install the Raspberry Pi OS (64-bit) on an SD card. The Raspberry Pi Imager also allows to setup a user account, Wi-Fi credentials, and enables an SSH server which is useful since we will be using this kit as a headless edge device.
After the installation is completed, we can insert the SD card back into the kit and power it on. Once it boots up, we can log in via ssh. The first thing we need to check, is whether the Akida PCIe card is detected or not using the command below which should print some information about the device.
To install the kernel driver and python bindings execute the following commands:
Execute the command below to make sure everything is okay and we can start using the PCIe card:
Shibuya Scramble Crossing is a popular pedestrian scramble crossing in Shibuya, Tokyo. There are several IP cameras installed at the crossing, and their live feeds are available on YouTube. We will be utilizing one of the streams for data collection.
To download the live video stream we need to install the youtube-dl python library.
We should be able to download a 640x360 pixels live stream using the command below.
We downloaded a few streams at different points in time during the afternoon and evening. We used the following Python script to extract every 30th frames from the video to remove redundancy.
We will use the Edge Impulse Studio to build and train our model. This requires us to create an account and initiate a new project at https://studio.edgeimpulse.com.
To upload the input images extracted from the video into the Edge Impulse project, we will use the Edge Impulse CLI Uploader. Follow the instructions at the link: https://docs.edgeimpulse.com/docs/cli-installation to install the Edge Impulse CLI on your host computer.
Execute the command below to upload the dataset.
The command above will upload the demo input images to Edge Impulse Studio and split them into "Training" and "Testing" datasets. Once the upload completes, the input datasets are visible on the Data Acquisition page within Edge Impulse Studio.
We can now assign labels to the data by using bounding boxes in the Labeling queue tab, as demonstrated in the GIF below. We have successfully labeled over 1800 objects, which was a tedious and time-consuming task, but it will greatly contribute to the creation of a diverse training dataset.
Go to the Impulse Design > Create Impulse page, click Add a processing block, and then choose Image. This preprocesses and normalizes image data, and optionally allows you to choose the color depth. Also, on the same page, click Add a learning block, and choose Object Detection (Images) - BrainChip Akida™ which fine-tunes a pre-trained object detection model specialized for the BrainChip AKD1000 PCIe board. This specialized model permits the use of a 224x224 image size, which is the size we are currently utilizing. Now click on the Save Impulse button.
On the Image page, choose RGB color depth and click the Save parameters button. The page will be redirected to the Generate Features page.
Now we can start feature generation by clicking on the Generate features button:
After feature generation, go to the Object Detection page and select the Akida FOMO model. Since we will be doing the inferencing on multiple camera streams we need to tweak the model to make it faster. We need to open the editor by selecting Switch to Keras (expert) mode to change the model parameters.
We changed the pre-trained model weights to the akidanet_imagenet_224_alpha_25.h5 and the train function parameter alpha to 0.25. The alpha parameter controls the width of the network. This is known as the width multiplier in the MobileNet paper. If alpha < 1.0, the number of filters in each layer proportionally decreases.
Then click on the Start training button. It will take a few minutes to complete the training.
Once the training is completed we can see the confusion matrix for the quantized Akida model as shown below.
On the Model testing page, click on the Classify All button which will initiate model testing with the trained model. The testing accuracy is 94.85%.
In order to achieve full hardware acceleration models must be converted from their original format to run on an Akida PCIe card. This can be done by selecting the BrainChip MetaTF Model from the Deployment page. This will generate a .zip file with models that can be used in the application for the AKD1000. The build process uses the CNN2SNN toolkit to convert quantized models to SNN models compatible with the AKD1000.
We will be using the Edge Impulse Linux C++ SDK which allows us to run machine learning model inferencing on Linux machines using C++. The SDK is open source and hosted on GitHub: edgeimpulse/example-standalone-inferencing-linux.
First, execute the command below to clone the repository.
We should extract the downloaded Brainchip MetaTF model zip file into the example-standalone-inferencing-linux directory.
We will not be using the OpenCV bundled with the repository instead we will install it from the OS distribution.
Also, we need to build and install an MPEG streamer library which is used to stream the inferencing result in a web browser.
For this application, we will be using one of the examples in the source directory, camera.cpp and modify it for our requirements. The complete code is given below.
We made the following changes in the Makefile to build it with the OpenCV library.
Finally, execute the command below which should build an executable camera in the build/ directory.
To run the application we should provide two recorded video files downloaded from the YouTube stream to the application executable as follows.
To view the inferencing results on a web page, we will save the following code in an index.html file.
We can spin up a webserver as follows and open a web browser at the port shown below.
We can see the real time stats for the inferencing rate and power consumption in the console.
In this project, we have evaluated the Brainchip AKD1000 Akida processor and demonstrated its effectiveness and efficiency in terms of accuracy, latency, bandwidth, and power consumption. We also conclude that Edge Impulse FOMO model is highly suitable for contrained and low-power edge devices to achieve fast inferencing without losing much accuracy. The public version of the Edge Impulse Studio project can be found here: https://studio.edgeimpulse.com/public/298672/latest.
