Count objects using FOMO

The Edge Impulse object detection model (FOMO) is effective at classifying objects and very lightweight (can run on MCUs). It does not however have any object persistence between frames. One common use of computer vision is for object counting- in order to achieve this you will need to add in some extra logic when deploying.

This notebook takes you through how to count objects using the linux deployment block (and provides some pointers for how to achieve similar logic other firmware deployment options).

Relevant links:

Raw python files for the linux deployment example: https://github.com/edgeimpulse/object-counting-demo
An end-to-end demo for on-device deployment of object counting: https://github.com/edgeimpulse/conveyor-counting-data-synthesis-demo

1. Download the linux deployment .eim for your project

To run your model locally, you need to deploy to a linux target in your project. First you need to enable all linux targets. Head to the deployment screen and click "Linux Boards" then in the following pop-up select "show all Linux deployment options on this page":

Then download the linux/mac target which is relevant to your machine:

Finally, follow the instructions shown as a pop-up to make your .eim file executable (for example for MacOS):

Open a terminal window and navigate to the folder where you downloaded this model.
Mark the model as executable: chmod +x path-to-model.eim
Remove the quarantine flag: xattr -d com.apple.quarantine ./path-to-model.eim

2. Object Detection

Dependencies

Ensure you have these libraries installed before starting:

! pip install edge_impulse_linux
! pip install numpy
! pip install opencv-python

2.1 Run Object Counting on a video file

(see next heading for running on a webcam)

This program will run object detection on an input video file and count the objects going upwards which pass a threshold (TOP_Y). The sensitivity can be tuned with the number of columns (NUM_COLS) and the DETECT_FACTOR which is the factor of width/height of the object used to determine object permanence between frames.

Ensure you have added the relevant paths to your model file and video file:

modelfile = '/path/to/modelfile.eim'
videofile = '/path/to/video.mp4'

import cv2
import os
import time
import sys, getopt
import numpy as np
from edge_impulse_linux.image import ImageImpulseRunner


modelfile = '/path/to/modelfile.eim'
videofile = '/path/to/video.mp4'


runner = None
# if you don't want to see a video preview, set this to False
show_camera = True
if (sys.platform == 'linux' and not os.environ.get('DISPLAY')):
    show_camera = False
print('MODEL: ' + modelfile)

with ImageImpulseRunner(modelfile) as runner:
    try:
        model_info = runner.init()
        print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
        labels = model_info['model_parameters']['labels']
        count = 0
        vidcap = cv2.VideoCapture(videofile)
        sec = 0
        start_time = time.time()

        def getFrame(sec):
            vidcap.set(cv2.CAP_PROP_POS_MSEC,sec*1000)
            hasFrames,image = vidcap.read()
            if hasFrames:
                return image
            else:
                print('Failed to load frame', videofile)
                exit(1)


        img = getFrame(sec)
        
    
        # Define the top of the image and the number of columns
        TOP_Y = 30
        NUM_COLS = 5
        COL_WIDTH = int(vidcap.get(3) / NUM_COLS)
        # Define the factor of the width/height which determines the threshold
        # for detection of the object's movement between frames:
        DETECT_FACTOR = 1.5

        # Initialize variables
        count = [0] * NUM_COLS
        countsum = 0
        previous_blobs = [[] for _ in range(NUM_COLS)]
        
        while img.size != 0:
            # imread returns images in BGR format, so we need to convert to RGB
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

            # get_features_from_image also takes a crop direction arguments in case you don't have square images
            features, cropped = runner.get_features_from_image(img)
            img2 = cropped
            COL_WIDTH = int(np.shape(cropped)[0]/NUM_COLS)

            # the image will be resized and cropped, save a copy of the picture here
            # so you can see what's being passed into the classifier
            cv2.imwrite('debug.jpg', cv2.cvtColor(cropped, cv2.COLOR_RGB2BGR))

            res = runner.classify(features)
            # Initialize list of current blobs
            current_blobs = [[] for _ in range(NUM_COLS)]
            
            if "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img2 = cv2.rectangle(cropped, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)

                        # Check which column the blob is in
                    col = int(bb['x'] / COL_WIDTH)

                    # Check if blob is within DETECT_FACTOR*h of a blob detected in the previous frame and treat as the same object
                    for blob in previous_blobs[col]:
                        if abs(bb['x'] - blob[0]) < DETECT_FACTOR * (bb['width'] + blob[2]) and abs(bb['y'] - blob[1]) < DETECT_FACTOR * (bb['height'] + blob[3]):
                        # Check this blob has "moved" across the Y threshold
                            if blob[1] >= TOP_Y and bb['y'] < TOP_Y:
                                # Increment count for this column if blob has left the top of the image
                                count[col] += 1
                                countsum += 1
                    # Add current blob to list
                    current_blobs[col].append((bb['x'], bb['y'], bb['width'], bb['height']))



            # Update previous blobs
            previous_blobs = current_blobs

            if (show_camera):
                im2 = cv2.resize(img2, dsize=(