LogoLogo
HomeDocsAPI & SDKsProjectsForumStudio
  • Welcome
    • Featured Machine Learning Projects
      • Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1
      • Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano
      • ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D
      • Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments
      • Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator
      • Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator
      • NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects
      • Community Guide – Using Edge Impulse with Nvidia DeepStream
      • Computer Vision Object Counting - Avnet RZBoard V2L
      • Gesture Appliances Control with Pose Detection - BrainChip AKD1000
      • Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)
      • High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)
    • Prototype and Concept Projects
      • Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning
      • TI CC1352P Launchpad - Getting Started with Machine Learning
      • OpenMV Cam RT1062 - Getting Started with Machine Learning
      • Getting Started with Edge Impulse Experiments
  • Computer Vision Projects
    • Workplace Organizer - Nvidia Jetson Nano
    • Recyclable Materials Sorter - Nvidia Jetson Nano
    • Analog Meter Reading - Arduino Nicla Vision
    • Creating Synthetic Data with Nvidia Omniverse Replicator
    • SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display
    • Traffic Monitoring - Brainchip Akida
    • Multi-camera Video Stream Inference - Brainchip Akida
    • Industrial Inspection Line - Brainchip Akida
    • X-Ray Classification and Analysis - Brainchip Akida
    • Inventory Stock Tracker - FOMO - BrainChip Akida
    • Container Counting - Arduino Nicla Vision
    • Smart Smoke Alarm - Arduino Nano 33
    • Shield Bot Autonomous Security Robot
    • Cyclist Blind Spot Detection - Himax WE-I Plus
    • IV Drip Fluid-Level Monitoring - Arduino Portenta H7
    • Worker PPE Safety Monitoring - Nvidia Jetson Nano
    • Delivered Package Detection - ESP-EYE
    • Bean Leaf Disease Classification - Sony Spresense
    • Oil Tank Measurement Using Computer Vision - Sony Spresense
    • Object Counting for Smart Industries - Raspberry Pi
    • Smart Cashier with FOMO - Raspberry Pi
    • PCB Defect Detection with Computer Vision - Raspberry Pi
    • Bicycle Counting - Sony Spresense
    • Counting Eggs with Computer Vision - OpenMV Cam H7
    • Elevator Passenger Counting - Arduino Nicla Vision
    • ESD Protection using Computer Vision - Seeed ReComputer
    • Solar Panel Defect Detection - Arduino Portenta H7
    • Label Defect Detection - Raspberry Pi
    • Dials and Knob Monitoring with Computer Vision - Raspberry Pi
    • Digital Character Recognition on Electric Meter System - OpenMV Cam H7
    • Corrosion Detection with Computer Vision - Seeed reTerminal
    • Inventory Management with Computer Vision - Raspberry Pi
    • Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L
    • Counting Retail Inventory with Computer Vision - Renesas RZ/V2L
    • Pose Detection - Renesas RZ/V2L
    • Product Quality Inspection - Renesas RZ/V2L
    • Smart Grocery Cart Using Computer Vision - OpenMV Cam H7
    • Driver Drowsiness Detection With FOMO - Arduino Nicla Vision
    • Gastroscopic Image Processing - OpenMV Cam H7
    • Pharmaceutical Pill Quality Control and Defect Detection
    • Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM
    • Smart Factory Prototype - Texas Instruments TDA4VM
    • Correct Posture Detection and Enforcement - Texas Instruments TDA4VM
    • Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM
    • Surface Crack Detection and Localization - Texas Instruments TDA4VM
    • Surface Crack Detection - Seeed reTerminal
    • Retail Image Classification - Nvidia Jetson Nano
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2
    • Object Detection and Visualization - Seeed Grove Vision AI Module
    • Bike Rearview Radar - Raspberry Pi
    • Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision
    • "Bring Your Own Model" Image Classifier for Wound Identification
    • Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano
    • Hardhat Detection in Industrial Settings - Alif Ensemble E7
    • Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A
    • Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A
    • Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense
    • Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5
    • Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit
    • Visual Anomaly Detection - Seeed Grove Vision AI Module V2
    • Object Counting with FOMO - OpenMV Cam RT1062
    • Visitor Heatmap with FOMO Object Detection - Jetson Orin Nano
    • Vehicle Security Camera - Arduino Portenta H7
  • Audio Projects
    • Occupancy Sensing - SiLabs xG24
    • Smart Appliance Control Using Voice Commands - Nordic Thingy:53
    • Glass Window Break Detection - Nordic Thingy:53
    • Illegal Logging Detection - Nordic Thingy:53
    • Illegal Logging Detection - Syntiant TinyML
    • Wearable Cough Sensor and Monitoring - Arduino Nano 33 BLE Sense
    • Collect Data for Keyword Spotting - Raspberry Pi Pico
    • Voice-Activated LED Strip - Raspberry Pi Pico
    • Snoring Detection on a Smart Phone
    • Gunshot Audio Classification - Arduino Nano 33 + Portenta H7
    • AI-Powered Patient Assistance - Arduino Nano 33 BLE Sense
    • Acoustic Pipe Leakage Detection - Arduino Portenta H7
    • Location Identification using Sound - Syntiant TinyML
    • Environmental Noise Classification - Nordic Thingy:53
    • Running Faucet Detection - Seeed XIAO Sense + Blues Cellular
    • Vandalism Detection via Audio Classification - Arduino Nano 33 BLE Sense
    • Predictive Maintenance Using Audio Classification - Arduino Nano 33 BLE Sense
    • Porting an Audio Project from the SiLabs Thunderboard Sense 2 to xG24
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 1
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 2
    • Keyword Spotting - Nordic Thingy:53
    • Detecting Worker Accidents with Audio Classification - Syntiant TinyML
    • Snoring Detection with Syntiant NDP120 Neural Decision Processor - Arduino Nicla Voice
    • Recognize Voice Commands with the Particle Photon 2
    • Voice Controlled Power Plug with Syntiant NDP120 (Nicla Voice)
    • Determining Compressor State with Audio Classification - Avnet RaSynBoard
    • Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline
    • Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier
  • Predictive Maintenance and Defect Detection Projects
    • Predictive Maintenance - Nordic Thingy:91
    • Brushless DC Motor Anomaly Detection
    • Industrial Compressor Predictive Maintenance - Nordic Thingy:53
    • Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense
    • Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal
    • Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal
    • Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal
    • Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549
    • BrickML Demo Project - 3D Printer Anomaly Detection
    • Condition Monitoring - Syntiant TinyML Board
    • Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense
    • Vibration Classification with BrainChip's Akida
    • AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32
  • Accelerometer and Activity Projects
    • Arduino x K-Way - Outdoor Activity Tracker
    • Arduino x K-Way - Gesture Recognition for Hiking
    • Arduino x K-Way - TinyML Fall Detection
    • Posture Detection for Worker Safety - SiLabs Thunderboard Sense 2
    • Hand Gesture Recognition - OpenMV Cam H7
    • Arduin-Row, a TinyML Rowing Machine Coach - Arduino Nicla Sense ME
    • Fall Detection using a Transformer Model – Arduino Giga R1 WiFi
    • Bluetooth Fall Detection - Arduino Nano 33 BLE Sense
    • Monitor Packages During Transit with AI - Arduino Nano 33 BLE Sense
    • Smart Baby Swing - Arduino Portenta H7
    • Warehouse Shipment Monitoring - SiLabs Thunderboard Sense 2
    • Gesture Recognition - Bangle.js Smartwatch
    • Gesture Recognition for Patient Communication - SiLabs Thunderboard Sense 2
    • Hospital Bed Occupancy Detection - Arduino Nano 33 BLE Sense
    • Porting a Posture Detection Project from the SiLabs Thunderboard Sense 2 to xG24
    • Porting a Gesture Recognition Project from the SiLabs Thunderboard Sense 2 to xG24
    • Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53
    • Classifying Exercise Activities on a BangleJS Smartwatch
  • Air Quality and Environmental Projects
    • Arduino x K-Way - Environmental Asthma Risk Assessment
    • Gas Detection in the Oil and Gas Industry - Nordic Thingy:91
    • Smart HVAC System with a Sony Spresense
    • Smart HVAC System with an Arduino Nicla Vision
    • Indoor CO2 Level Estimation - Arduino Portenta H7
    • Harmful Gases Detection - Arduino Nano 33 BLE Sense
    • Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense
    • AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3
    • AI-Assisted Air Quality Monitoring - DFRobot Firebeetle ESP32
    • Air Quality Monitoring with Sipeed Longan Nano - RISC-V Gigadevice
    • Methane Monitoring in Mines - Silabs xG24 Dev Kit
    • Smart Building Ventilation with Environmental Sensor Fusion
    • Sensor Data Fusion with Spresense and CommonSense
    • Water Pollution Detection - Arduino Nano ESP32 + Ultrasonic Scan
    • Fire Detection Using Sensor Fusion - Arduino Nano 33 BLE Sense
  • Novel Sensor Projects
    • 8x8 ToF Gesture Classification - Arduino RP2040 Connect
    • Food Irradiation Dose Detection - DFRobot Beetle ESP32C3
    • Applying EEG Data to Machine Learning, Part 1
    • Applying EEG Data to Machine Learning, Part 2
    • Applying EEG Data to Machine Learning, Part 3
    • Liquid Classification with TinyML - Seeed Wio Terminal + TDS Sensor
    • AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar
    • Soil Quality Detection Using AI and LoRaWAN - Seeed Sensecap A1101
    • Smart Diaper Prototype - Arduino Nicla Sense ME
    • DIY Smart Glove with Flex Sensors
    • EdgeML Energy Monitoring - Particle Photon 2
    • Wearable for Monitoring Worker Stress using HR/HRV DSP Block - Arduino Portenta
  • Software Integration Demos
    • Azure Machine Learning with Kubernetes Compute and Edge Impulse
    • ROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python
    • ROS2 + Edge Impulse, Part 2: MicroROS
    • Using Hugging Face Datasets in Edge Impulse
    • Using Hugging Face Image Classification Datasets with Edge Impulse
    • Edge Impulse API Usage Sample Application - Jetson Nano Trainer
    • MLOps with Edge Impulse and Azure IoT Edge
    • A Federated Approach to Train and Deploy Machine Learning Models
    • DIY Model Weight Update for Continuous AI Deployments
    • Automate the CI/CD Pipeline of your Models with Edge Impulse and GitHub Actions
    • Deploying Edge Impulse Models on ZEDEDA Cloud Devices
Powered by GitBook
On this page
  • Project Demo
  • Introduction
  • Texas Instruments SK-TDA4VM
  • RTSP Streaming
  • On to my Project
  • Compatible Models with SK-TDA4VM
  • PSDK Compatibilities
  • Scaling

Was this helpful?

Edit on GitHub
Export as PDF
  1. Computer Vision Projects

Smart Factory Prototype - Texas Instruments TDA4VM

Using a TI TDA4VM, RTSP streaming, and an Edge Impulse model, monitor the status on your factory equipment and take action when needed.

PreviousDeter Shoplifting with Computer Vision - Texas Instruments TDA4VMNextCorrect Posture Detection and Enforcement - Texas Instruments TDA4VM

Last updated 1 year ago

Was this helpful?

Created By: Justin Lutz

Public Project Link:

GitHub Repo:

Project Demo

Introduction

Knowing the status and health of factory machinery is critical to an organization's success. It takes factory operator vigilance to regularly monitor equipment and take action if anomalous behavior is detected. However, it can be fatiguing for personnel to constantly monitor equipment, and if an issue is missed, weeks of downtime for costly repairs can be the result.

This is where the power of computer vision on the edge can be invaluable. Using a computer model trained to detect nominal and off-nominal behavior, operators can be alerted of issues, rather than having to be constantly on the lookout. And with the inferencing being done at the edge, privacy is maintained, and organizational leadership can breathe easy that perhaps sensitive images won't be sent to the cloud for remote inferencing.

Texas Instruments SK-TDA4VM

I was lucky enough to receive a starter kit from the TI and Edge Impulse teams to integrate an Edge Impulse model onto the kit. I wanted to go a step further and test out RTSP streaming with the SK-TDA4VM.

RTSP Streaming

The next challenge was correctly forming the gstreamer pipeline to correctly receive the RTSP stream. That proved a bit of a challenge (for a gstreamer noob like me) and after much trial and error was able to run an RTSP stream on the SK-TDA4VM! Below is the gstreamer pipeline that worked for me for the RTSP streamer example:

./test-launch  --gst-debug=3 '( v4l2src device=/dev/video0 ! image/jpeg,width=640,height=480,framerate=30/1 ! jpegdec ! videoconvert ! video/x-raw,width=640,height=480,framerate=30/1 ! x264enc tune="zerolatency" byte-stream=true bitrate=2000 ! rtph264pay name=pay0 pt=96 )'

While I was eventually successful, this setup took days to figure out, and I hope TI has more out of the box solutions in the future.

On to my Project

Once I had the setup working, I moved the reComputer down to my basement for my project. Since I don't have any factory equipment in my house, I decided to use my heating zone controller as a demonstration of the capabilities of an Edge Impulse model with the SK-TDA4VM hardware and an RTSP stream.

Once all the training and test images were labeled, I created my impulse:

Since my input stream was 640 x 480, I decided to have my model image dimensions be 320 x 320 (they have to be square). I selected Image for the processing block, and Object Detection for the learning block.

Since I wanted to differentiate between the power LED (which is green) and the zone LEDs (which are red), I decided to use RGB images for my model. This ruled out FOMO, which is greyscale only. In hindsight, I think I could have done FOMO and just differentiate by bounding box coordinates which LED is power (it would always be at the top) and which is a zone LED; color wouldn't have to be a determining factor.

Compatible Models with SK-TDA4VM

Once I had that custom block created, I retrained my model with YoloX:

I ran against the test data with great results:

I then deployed my model as a TI TIDL-RT library.

PSDK Compatibilities

The output of library build is a zip file, however, the contents are in the format for the edge-impulse-linux-runner, and not the TI model zoo format that's found in the SDK. I organized the model contents into a similar format as the other models in the model zoo:

Scaling

With the release of the Edge Impulse TIDL-RT for PSDK v8.05, I was now able to load the Edge Impulse model and see the RTSP stream without the application crashing. However, I was seeing HUNDREDS of detections. With the help of the Edge Impulse SMEs, I learned that there is a difference between the image input range of the Edge Impulse SDK (0...1) and the YoloX input range (0...255). I therefore had to scale the image input to 1/255 to feed into the Edge Impulse SDK. After a couple hours of searching, I found a variable in the model's params.yaml file that can scale the input:

input_scale:
    - 0.0039215697906911373
    - 0.0039215697906911373
    - 0.0039215697906911373

This was originally set to 1.0. If you do the quick math, 1.0 / 255.0 = 0.0039215697906911373. After I had changed that scaling factor, success!

As you can see below, I was seeing pretty solid inference times of 50 ms for ~20 fps at 640x480 over RTSP. Not bad! I think the 50 ms inference is really strong for object detection.

If you look at my GitHub, you can see my settings for the rtsp_src_example.yaml, as well as the model contents and configurations. I made some small changes to the post_process.py python file as well.

Overall, this was a good test of learning new concepts for development. From getting the RTSP server up and running, to creating custom blocks, to ensuring the model inputs aligned, I came across several obstacles that I didn't anticipate. However, the end result was a solid demonstration of the SK-TDA4VM board when paired with an Edge Impulse object detection model.

The TI SK-TDA4VM starter kit is up to the task for smart factory operations. With the and 4GB of LPDDR4‐4266 RAM, the system can execute up to 8 Trillion Operations Per Second (TOPS). A full overview of the hardware specs, along with links to the startup guide, .

I first started by following and downloading the OS and Processor SDK onto an SD card. At the time of this writing, that version was 8.05. After writing the SDK to a micro SD card, connecting a small HDMI monitor, and connecting a micro USB cable to the UART port, I was up and running fairly quickly and seeing boot up output through my putty serial connection. However, I don't have an Ethernet router near my office (I'm pretty much all wireless) so I ordered an card to get the ability to connect to the Internet (and receive RTSP streams). I followed and connected essentially out of the box. Once I had the internet connection, I remotely connected through Visual Studio Code (as described in the startup guide) so I could edit files through VS Code instead of VI (which I highly recommend).

Since I was planning on running an RTSP stream, I didn't connect a USB or Raspberry Pi camera to the board. Before getting into the RTSP streaming though, I started experimenting with the sample models that come with the SDK, for example image classification, object detection, image segmentation, and pose estimation. I using the Python demo scripts and found it pretty straightforward to use the config yaml files to change models, inputs, and outputs.

Before building my Edge Impulse model, I wanted to ensure that I could get RTSP streaming up and running, which proved to be a bit of a challenge. I've had using a commercial RTSP streaming camera out of the box, but they were not working with the SK-TDA4VM. I reached out to TI's E2E technical forum, and they were prompt in responding to my issues. My thread on the E2E forum can be . I ended up using my Seeed Studio reComputer 1010 (with a Jetson Nano) with an inexpensive USB camera as my setup for the RTSP streamer. Following the instructions of the tutorial, I was able to get the test source working.

Once I had the setup complete, and the RTSP server running, I starting collecting data for my Edge Impulse model. For that I simply used VLC media player to stream the RTSP stream. If would then click on Video > Take Snapshot to take a screenshot of my controller. I would use my smartphone to change the temperature in various zones, which would cause those zone lights to illuminate red on the controller board. I took about a hundred photos with various lights illuminated (and off). Ideally you would have several hundred images but this was a good start. Once I had those images collected, it was time to start up the and start labelling data. I elected to do an object detection model since I wanted to identify the number of zones illuminated (and hence on), so labelling consisted of created bounding boxes around the illuminated zones (and power LED).

I initially selected a YoloV5 model as the model block for my transfer learning model, but I soon found out after deploying it that it wasn't compatible with the TI Deep Learning (TIDL) AI accelerators. Thanks to the Edge Impulse team, I learned that I needed to that would play nice with the TIDL framework. This tutorial took me a bit to complete as your system needs to be configured for Virtualization (which mine wasn't) and Docker desktop needs to be installed. It took a couple nights of trying, but I was finally able to push my custom YoloX block to the project.

A public version of the project can be .

However, I received error messages trying to load the model. After another , I found out that the Edge Impulse model library was for the 8.04 SDK, and I was running version 8.05 and the two were thus incompatible. Fortunately for me, within days, Edge Impulse had up-rev'd their library output to be compatible with 8.05, and another hurdle was cleared!

TDA4VM processor
can be found here
the startup guide
Intel 9260 Wireless PCIe M2e
this guide from TI
followed this tutorial
success on other projects
found here
Edge Impulse Studio
use a TI fork of the YoloX model
found here
trip to TI E2E
https://studio.edgeimpulse.com/public/187321/latest
https://github.com/jlutzwpi/SK-TDA4VM
My setup with the Intel Wireless card installed and antennas connected.
RTSP streaming success on the SK-TDA4VM, with a Seeed Studio reComputer 1010 as the RTSP server
USB camera pointing towards the controller and RTSP server running on Seeed Studio reComputer.
Ingested images with labels and bounding boxes applied
Training with the TI fork of YoloX for my model.
Results with the test data looked good.
50ms inference at ~20 fps over an RTSP stream, not bad!