Services

SDK documentation

Machine learning (ML) is a way of writing computer programs. Specifically, it’s a way of writing programs that process raw data and turn it into information that is meaningful at an application level.

For example, one ML program might be designed to determine when an industrial machine has broken down based on readings from its various sensors, so that it can alert the operator. Another ML program might take raw audio data from a microphone and determine if a word has been spoken, so it can activate a smart home device.

Unlike normal computer programs, the rules of ML programs are not determined by a developer. Instead, ML uses specialized algorithms to learn rules from data, in a process known as training.

In a traditional piece of software, an engineer designs an algorithm that takes an input, applies various rules, and returns an output. The algorithm’s internal operations are planned out by the engineer and implemented explicitly through lines of code. To predict breakdowns in an industrial machine, the engineer would need to understand which measurements in the data indicate a problem and write code that deliberately checks for them.

This approach works fine for many problems. For example, we know that water boils at 100°C at sea level, so it’s easy to write a program that can predict whether water is boiling based on its current temperature and altitude. But in many cases, it can be difficult to know the exact combination of factors that predicts a given state. To continue with our industrial machine example, there might be various different combinations of production rate, temperature, and vibration level that might indicate a problem but are not immediately obvious from looking at the data.

To create an ML program, an engineer first collects a substantial set of training data. They then feed this data into a special kind of algorithm, and let the algorithm discover the rules. This means that as ML engineers, we can create programs that make predictions based on complex data without having to understand all of the complexity ourselves.

Through the training process, the ML algorithm builds a model of the system based on the data we provide. We run data through this model to make predictions, in a process called inference.

Where can machine learning help?

Machine learning is an excellent tool for solving problems that involve pattern recognition, especially patterns that are complex and might be difficult for a human observer to identify. ML algorithms excel at turning messy, high-bandwidth raw data into usable signals, especially combined with conventional signal processing.

For example, the average person might struggle to recognize the signs of a machine failure given ten different streams of dense, noisy sensor data. However, a machine learning algorithm can often learn to spot the difference.

But ML is not always the best tool for the job. If the rules of a system are well defined and can be easily expressed with hard-coded logic, it’s usually more efficient to work that way.

What is embedded ML?

Recent advances in microprocessor architecture and algorithm design have made it possible to run sophisticated machine learning workloads on even the smallest of microcontrollers. Embedded machine learning, also known as TinyML, is the field of machine learning when applied to embedded systems such as these.

Bandwidth—ML algorithms on edge devices can extract meaningful information from data that would otherwise be inaccessible due to bandwidth constraints.

Latency—On-device ML models can respond in real-time to inputs, enabling applications such as autonomous vehicles, which would not be viable if dependent on network latency.

Economics—By processing data on-device, embedded ML systems avoid the costs of transmitting data over a network and processing it in the cloud.

Reliability—Systems controlled by on-device models are inherently more reliable than those which depend on a connection to the cloud.

Privacy—When data is processed on an embedded system and is never transmitted to the cloud, user privacy is protected and there is less chance of abuse.

Learn more

After creating your Edge Impulse Studio project, you will be directed to the project's dashboard. The dashboard gives a quick overview of your project such as your project ID, the number of devices connected, the amount of data collected, the preferred labeling method, among other editable properties. You can also enable some additional capabilities to your project such as collaboration, making your project public, and showcasing your public projects using Markdown READMEs as we will see.

The figure below shows the various sections and widgets of the dashboard that we will cover here.

1. Getting Started

The Getting Started section is here to help. You can choose from 3 different options to get started:

• Add existing data: When selecting this option, you can then choose to Upload data from your computer or to Add storage bucket

• Collect new data: When selecting this option, the getting started guide will ask you to either Scan QR code to connect to your phone, Connect to your computer, or to Connect your device or developer board. Make sure that your device or development board is flashed with the Edge Impulse official firmware.

2. Making your project public

To share your private project with the world, and click Make this project public.

By doing this, all of your data, block configurations, intermediate results, and final models will be shared with the world. Your project will be publicly accessible and can be cloned with a single click with the provided URL:

3. Run this model

When you have a trained model available in your project, this card will appear to let you test your model with your phone or your computer. This is particularly useful to validate the behaviour of your model before integrating it into your targeted embedded firmware.

4. Collaboration

To add a collaborator, go to your project's dashboard and find the "Collaborators" widget. Click the '+' icon and type the username or e-mail address of the other user. The user will be invited to create an Edge Impulse account if it doesn't exist.

The user will be automatically added to the project and will get an email notification inviting them to start contributing to your project. To remove a user, simply click on the three dots besides the user then tap ‘Delete’ and they will be automatically removed.

5. Showcasing your public projects with Markdown READMEs

The project README enables you to explain the details of your project in a short way. Using this feature, you can add visualizations such as images, GIFs, code snippets, and text to your project in order to bring your colleagues and project viewers up to speed with the important details of your project. In your README you might want to add things like:

• What the project does

• Why the project is useful

• Motivations of the project

• How to get started with the project

• What sensors and target deployment devices you used

• How you plan to improve your project

• Where users can get help with your project

To create your first README, navigate to the "about this project" widget and click "add README"

For more README inspiration, check out the public Edge Impulse project tutorials below:

6. Project info

The project info widget shows the project's specifications such as the project ID, labeling method, and latency calculations for your target device.

• On the labeling method dropdown, you need to specify the type of labeling your dataset and model expect. This can be either one label per data item or bounding boxes. Bounding boxes only work for object detection tasks in the studio. Note that if you interchange the labeling methods, learning blocks will appear to be hidden when building your impulse.

• One of the amazing Edge Impulse superpowers is the latency calculation component. This is an approximate time in milliseconds that the trained model and DSP operations are going to take during inference based on the selected target device. This hardware in the loop approach ensures that the target deployment device compute resources are not underutilized or over-utilized. It also saves developers' time associated with numerous inference iterations back and forth the studio in search of optimum models.

7. Block Outputs

In the Block Output section, you can download the results of the DSP and ML operations of your impulse.

The downloadable assets include the extracted features, Tensorflow SavedModel, and both quantized and unquantized TensorFlow lite models. This is particularly helpful when you want to perform other operations to the output blocks outside the Edge Impulse studio. For example, if you need a TensorflowJS model, you will just need to download the TensorFlow saved model from the dashboard and convert it to TensorFlowJS model format to be served on a browser.

8. Performance Settings

This section consists of editable parameters that directly affect the performance of the studio when building your impulse. Depending on the selected or available settings, your jobs can either be fast or slow.

The use of GPU for training and Parallel DSP jobs is currently an internal experimental feature that will be soon released.

9. Administrative Zone

To bring even more flexibility to projects, the administrative zone gives developers the power to enable other additional features that are not found in edge impulse projects by default. Most of these features are usually advanced features intended for organizations or sometimes experimental features.

To activate these features you just need to check the boxes against the specific features you want to use and click save experiments.

10. Danger Zone

The danger zone widget consists of irrevocable actions that let you to:

• Perform train/test split. This action re-balances your dataset by splitting all your data automatically between the training and testing set and resets the categories for all data.

• Launch the getting started wizard. This will remove all data, and clear out your impulse.

• Transfer ownership. This action is available for users who have one or more organizations linked with their accounts. With it, you can start working on a project with your user profile and then transfer the ownership to your organization.

• Delete your project. This action removes all devices, data, and impulses from your project.

• Delete all data in this project.

Congratulations, you've trained your first embedded machine learning model! This page lists next steps you can take to make your devices smarter.

Run your model on a real device

More than audio

Making a machine learning model that responds to your voice is cool, but you can do a lot more with Edge Impulse. Here are a number of tutorials to get you started:

Make your model more robust by adding more data

Your model was trained on +/- 20 seconds of data, which is a very small amount of data. To make your model more robust you can add more data.

• If your model does not respond well enough on your keyword (e.g. if you have someone saying the word in a different tone or pitch), record some more data of the keyword.

• If the model is too sensitive (triggers when you say something else), then say some different words and label them with the 'unknown' class.

Share your project with the world

Think your model is awesome, and want to share it with the world? Go to Dashboard and click Make this project public. This will make your whole project - including all data, machine learning models and visualizations - available, and can be viewed and cloned by anyone with the URL.

More questions? Ask us on the forums!

On the Devices tab, you'll find a list of all your connected devices and a guide on how to connect new devices that are currently supported by Edge Impulse.

To connect a new device, click on the Connect a new device button on the top right of your screen.

You will get a pop-up with multiple options of devices you can connect to your Edge Impulse project. Available options include:

How can I share my Edge Impulse project?

You can also create a public version of your Edge Impulse project. This makes your project available to the whole world - including your data, your impulse design, your models, and all intermediate information - and can easily be cloned by anyone in the community. To do so, go to Dashboard, and click Make this project public.

What are the minimum hardware requirements to run the Edge Impulse inferencing library on my embedded device?

What frameworks does Edge Impulse use to train the machine learning models?

We use a wide variety of tools, depending on the machine learning model. For neural networks we typically use TensorFlow and Keras, for object detection models we use TensorFlow with Google's Object Detection API, and for 'classic' non-neural network machine learning algorithms we mainly use sklearn. For neural networks you can see (and modify) the Keras code by clicking ⋮, and selecting Switch to expert mode.

What engine does Edge Impulse use to compile the Impulse?

It depends on the hardware.

For general-purpose MCUs we typically use EON Compiler with TFLite Micro kernels (including hardware optimization, e.g. via CMSIS-NN, ESP-NN).

On Linux, if you run the Impulse on CPU, we use TensorFlow Lite.

For accelerators we use a wide variety of other runtimes, e.g. hardcoded network in silicon for Syntiant, custom SNN-based inference engine for Brainchip Akida, DRP-AI for Renesas RZV2L, etc...

Is there a downside to enabling the EON Compiler?

By disabling EON we place the full neural network (architecture and weights) into ROM, and load it on demand. This increases memory usage, but you could just update this section of the ROM (or place the neural network in external flash, or on an SD card) to make it easier to update.

Can I use a model that has been trained elsewhere in Edge Impulse?

How does the feature explorer visualize data that has more that 3 dimensions?

Does Edge impulse integrate with other cloud services?

Yes. The enterprise version of Edge Impulse can integrate directly with your cloud service to access and transform data.

What is the typical power consumption of the Edge Impulse machine learning processes on my device?

Simple answer: To get an indication of time per inference we show performance metrics in every DSP and ML block in the Studio. Multiply this by the active power consumption of your MCU to get an indication of power cost per inference.

More complicated answer: It depends. Normal techniques to conserve power still apply to ML, so try to do as little as possible (do you need to classify every second, or can you do it once a minute?), be smart about when to run inference (can there be an external trigger like a motion sensor before you run inference on a camera?), and collect data in a lower power mode (don't run at full speed when sampling low-resolution data, and see if your sensor can use an interrupt to wake your MCU - rather than polling).

What is the .eim model format for Edge Impulse for Linux?

How is the labeling of the data performed?

Can I use an unsupported development board or a custom PCB (with a different microcontroller or microprocessor) with Edge Impulse?

Yes! A "supported board" simply means that there is an official or community-supported firmware that has been developed specifically for that board that helps you collect data and run impulses. Edge Impulse is designed to be extensible to computers, smartphones, and a nearly endless array of microcontroller build systems.

You can collect data and upload it to Edge Impulse in a variety of ways. For example:

In object detection ML projects, labeling is the process of defining regions of interest in the frame.

Manually labeling images can become tedious and time-consuming, especially when dealing with huge datasets. This is why Edge Impulse studio provides an AI-assisted labeling tool to help you in your labeling workflows.

To use the labeling queue, you will need to set your Edge Impulse project as an "object detection" project. The labeling queue will only display the images that have not been labeled.

Currently, it only works to define bounding boxes (ingestion format used to train both MobileNetv2 SSD and FOMO models).

AI-Assisted labeling

There are 3 ways you can use to perform AI-assisted labeling on the Edge Impulse Studio:

• Using yolov5

• Using your own model

• Using object tracking

Using YOLOv5

By utilizing an existing library of pre-trained object detection models from YOLOv5 (trained with the COCO dataset), common objects in your images can quickly be identified and labeled in seconds without needing to write any code!

To label your objects with YOLOv5 classification, click the Label suggestions dropdown and select “Classify using YOLOv5.” If your object is more specific than what is auto-labeled by YOLOv5, e.g. “coffee” instead of the generic “cup” class, you can modify the auto-labels to the left of your image. These modifications will automatically apply to future images in your labeling queue.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes!

Using your own model

You can also use your own trained model to predict and label your new images. From an existing (trained) Edge Impulse object detection project, upload new unlabeled images from the Data Acquisition tab.

From the “Labeling queue”, click the Label suggestions dropdown and select “Classify using ”:

You can also upload a few samples to a new object detection project, train a model, then upload more samples to the Data Acquisition tab and use the AI-Assisted Labeling feature for the rest of your dataset. Classifying using your own trained model is especially useful for objects that are not in YOLOv5, such as industrial objects, etc.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes using your own pre-trained model!

Using Object tracking

If you have objects that are a similar size or common between images, you can also track your objects between frames within the Edge Impulse Labeling Queue, reducing the amount of time needed to re-label and re-draw bounding boxes over your entire dataset.

Draw your bounding boxes and label your images, then, after clicking Save labels, the objects will be tracked from frame to frame:

Now that your object detection project contains a fully labeled dataset, learn how to train and deploy your model to your edge device: check out our tutorial!

We are excited to see what you build with the AI-Assisted Labeling feature in Edge Impulse, please post your project on our forum or tag us on social media, @Edge Impulse!

Get started with any device

Follow these three steps to build your first embedded Machine Learning model - no worries, you can use almost any device to get started.

1. You'll need some data:
3. After training your model you can run your model on your device:

   • If you have a fully supported development board (or your mobile phone) you can build new firmware - which includes your model - directly from the UI. It doesn't get easier than that!

Suitable for any type of embedded ML application

API Documentation

Enterprise version

The data explorer is a visual tool to explore your dataset, find outliers or mislabeled data, and to help label unlabeled data. The data explorer first tries to extract meaningful features from your data (through signal processing and neural network embeddings) and then uses a dimensionality reduction algorithm to map these features to a 2D space. This gives you a one-look overview of your complete dataset.

Using the data explorer

To access the data explorer head to Data acquisition, click Data explorer, then select a way to generate the data explorer. Depending on you data you'll see three options:

• Using a pre-trained model - here we use a large neural network trained on a varied dataset to generate the embeddings. This works very well if you don't have any labeled data yet, or want to look at new clusters of data. This option is available for keywords and for images.

• Using your trained impulse - here we use the neural network block in your impulse to generate the embeddings. This typically creates even better visualizations, but will fail if you have completely new clusters of data as the neural network hasn't learned anything about them. This option is only available if you have a trained impulse.

Then click Generate data explorer to create the data explorer. If you want to make a different choice after creating the data explorer click ⋮ in the top right corner and select Clear data explorer.

Viewing and modifying data

To view an item in your dataset just click on any of the dots (some basic information appears on hover). Information about the sample, and a preview of the data item appears at the bottom of the data explorer. You can click Set label (or l on your keyboard) to set a new label for the data item, or press Delete item (or d on your keyboard) to remove the data item. These changes are queued until you click Save labels (at the top of the data explorer).

Assisted labeling

The data explorer marks unlabeled data in gray (with an 'Unlabeled' label). To label this data you click on any gray dot. To then set a label by clicking the Set label button (or by pressing l on your keyboard) and enter a label. Other unlabeled data in the vicinity of this item will automatically be labeled as well. This way you can quickly label clustered data.

To upload unlabeled data you can either:

• Select the items in your dataset under Data acquisition, select all relevant items, click Edit labels and set the label to an empty string.

Or, if you want to start from scratch, click the three dots on top of the data explorer, and select Clear all labels.

Wait, how does this work?

The data explorer uses a three-stage process:

1. It runs your data through an input and a DSP block - like any impulse.

2. It passes the result of 1) through part of a neural network. This forces the neural network to compress the DSP output even further, but to features that are highly specialized to distinguish the exact type of data in your dataset (called 'embeddings').

3. The embeddings are passed through t-SNE, a dimensionality reduction algorithm.

• 33 input features (from the signal processing step)

• A layer with 20 neurons

• A layer with 10 neurons

• A layer with 4 neurons (the number of different classes)

While training the neural network we try to find the mathematical formula that best maps the input to the output. We do this by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step - just like our signal processing step - but highly tuned for your specific data. For example, in the first layer, it'll learn what features are correlated, in the second it derives new features, and in the final layer, it learns how to distinguish between classes of motions.

In the data explorer we now cut off the final layer of the neural network, and thus we get the derived features back - these are called "embeddings". Contrary to features we extract using signal processing we don't really know what these features are - they're specific to your data. In essence, they provide a peek into the brain of the neural network. Thus, if you see data in the data explorer that you can't easily separate, the neural network probably can't either - and that's a great way to spot outliers - or if there's unlabeled data close to a labeled cluster they're probably very similar - great for labeling unknown data!

Examples of different embeddings

Here's an example of using the data explorer to visualize a very complex computer vision dataset (distinguishing between the four cats of one of our infrastructure engineers).

No embeddings (just running t-SNE over the images)

With embeddings from a pre-trained MobileNetV2 model

With embeddings from a custom ML model

For less complex datasets, or lower-dimensional data you'll typically see more separation, even without custom models.

Questions? Excited?

You can upload your existing data samples and datasets to your project directly through the Edge Impulse Studio Uploader.

The uploader currently handles these types of files:

• .wav - Lossless audio files. It's recommended to use the same frequency for all files in your data set, as signal processing output might be dependent on the frequency.

• .jpg and .png - Image files. It's recommended to use the same ratio for all files in your data set.

• .mp4 and .avi- Video file. You can then from the studio split this video file into images at a configurable frame per second.

To upload data using the uploader, go to the Data acquisition page and click on the uploader button as shown in the image below:

Upload data

Upload mode

Select individual files: This option let you select multiple individual files within a single folder. If you want to upload images with bounding boxes, make sure to also select the label files.

Select a folder: This option let you select one folder, including all the subfolders.

Upload into a category

Select which category you want to upload your dataset into. Options can be training, testing or perform an 80/20 split between your data samples.

Label

Dataset annotation format

Image datasets can be found in a range of different formats. Different formats have different directory structures, and require annotations (or labels) to follow a particular structure. We support uploading data in many different formats in the Edge Impulse Studio.

Image datasets usually consist of a bunch of image files, and one (or many) annotation files, which provide labels for the images. Image datasets may have annotations that consist of:

• A single-label: each image has a single label

• Bounding boxes: used for object detection; images contain 'objects' to be detected, given as a list of labeled 'bounding boxes'

When you upload an image dataset, we try to automatically detect the format of that data (in some cases, we cannot detect it and you will need to manually select it).

Once the format of your dataset has been selected, click on Upload Data and let the Uploader parse your dataset:

Understanding image dataset annotation formats

Unlabeled

Leave the data unlabeled, you can manually label your data sample in the studio.

Edge Impulse object detection format

The Edge Impulse object detection acquisition format provides a simple and intuitive way to store images and associated bounding box labels. Folders containing data in this format will take the following structure:

The subdirectories contain image files in JPEG or PNG format. Each image file represents a sample and is associated with its respective bounding box labels in the bounding_boxes.labels file.

The bounding_boxes.labels file in each subdirectory provides detailed information about the labeled objects and their corresponding bounding boxes. The file follows a JSON format, with the following structure:

• version: Indicates the version of the label format.

• files: A list of objects, where each object represents an image and its associated labels.

  • path: The path or file name of the image.

  • category: Indicates whether the image belongs to the training or testing set.

  • (optional) label: Provides information about the labeled objects.

    • type: Specifies the type of label (e.g., a single label).

    • label: The actual label or class name of the object.

  • (Optional) metadata: Additional metadata associated with the image, such as the site where it was collected, the timestamp or any useful information.

  • boundingBoxes: A list of objects, where each object represents a bounding box for an object within the image.

    • label: The label or class name of the object within the bounding box.

    • x, y: The coordinates of the top-left corner of the bounding box.

    • width, height: The width and height of the bounding box.

bounding_boxes.labels example:

COCO JSON

The COCO JSON (Common Objects in Context JSON) format is a widely used standard for representing object detection datasets. It provides a structured way to store information about labeled objects, their bounding boxes, and additional metadata.

A COCO JSON dataset can follow this directory structure:

The _annotations.coco.json file in each subdirectory provides detailed information about the labeled objects and their corresponding bounding boxes. The file follows a JSON format, with the following structure:

Categories

The "categories" component defines the labels or classes of objects present in the dataset. Each category is represented by a dictionary containing the following fields:

• id: A unique integer identifier for the category.

• name: The name or label of the category.

• (Optional) supercategory: A higher-level category that the current category belongs to, if applicable. This supercategory is not used or imported by the Uploader.

Images

The "images" component stores information about the images in the dataset. Each image is represented by a dictionary with the following fields:

• id: A unique integer identifier for the image.

• width: The width of the image in pixels.

• height: The height of the image in pixels.

• file_name: The file name or path of the image file.

Annotations

The "annotations" component contains the object annotations for each image. An annotation refers to a labeled object and its corresponding bounding box. Each annotation is represented by a dictionary with the following fields:

• id: A unique integer identifier for the annotation.

• image_id: The identifier of the image to which the annotation belongs.

• category_id: The identifier of the category that the annotation represents.

• bbox: A list representing the bounding box coordinates in the format [x, y, width, height].

• (Optional) area: The area (in pixels) occupied by the annotated object.

• (Optional) segmentation: The segmentation mask of the object, represented as a list of polygons.

• (Optional) iscrowd: A flag indicating whether the annotated object is a crowd or group of objects.

Edge Impulse uploader currently doesn't import the area, segmentation, iscrowd fields.

_annotations.coco.json example:

Open Images CSV

The OpenImage dataset provides object detection annotations in CSV format. The _annotations.csv file is located in the same directory of the images it references. A class-descriptions.csv mapping file can be used to give short description or human-readable classes from the MID LabelName.

An OpenImage CSV dataset usually has this directory structure:

Annotation Format:

• Each line in the CSV file represents an object annotation.

• The values in each line are separated by commas.

CSV Columns:

• The CSV file typically includes several columns, each representing different attributes of the object annotations.

• The common columns found in the OpenImage CSV dataset include:

  • ImageID: An identifier or filename for the image to which the annotation belongs.

  • Source: The source or origin of the annotation, indicating whether it was manually annotated or obtained from other sources.

  • LabelName: The class label of the object.

  • Confidence: The confidence score or probability associated with the annotation.

  • XMin, YMin, XMax, YMax: The coordinates of the bounding box that encloses the object, usually represented as the top-left (XMin, YMin) and bottom-right (XMax, YMax) corners.

  • IsOccluded, IsTruncated, IsGroupOf, IsDepiction, IsInside: Binary flags indicating whether the object is occluded, truncated, a group of objects, a depiction, or inside another object.

Currently, Edge Impulse only imports these fields:

Class Labels:

• Each object in the dataset is associated with a class label.

• The class labels in the OpenImage dataset are represented as LabelName in the CSV file.

• The LabelName correspond to specific object categories defined in the OpenImage dataset's ontology (MID).

Note that Edge Impulse does not enforce this ontology, if you have an existing dataset using the MID LabelName, simply provide a class-description.csv mapping file to see your classes in Edge Impulse Studio.

Bounding Box Coordinates:

• The bounding box coordinates define the normalized location and size of the object within the image.

• The coordinates are represented as the X and Y pixel values for the top-left corner (XMin, YMin) and the bottom-right corner (XMax, YMax) of the bounding box.

class-descriptions.csv mapping file:

• To be ingested in Edge Impulse the mapping file name must end with *class-descriptions.csv

_annotations.csv example:

Pascal VOC XML

The Pascal VOC (Visual Object Classes) format is another widely used standard for object detection datasets. It provides a structured format for storing images and their associated annotations, including bounding box labels.

A Pascal VOC dataset can follow this directory structure:

The Pascal VOC dataset XML format typically consists of the following components:

1. Image files: The dataset includes a collection of image files, usually in JPEG or PNG format. Each image represents a sample in the dataset.

2. Annotation files: The annotations for the images are stored in XML files. Each XML file corresponds to an image and contains the annotations for that image, including bounding box labels and class labels.

3. Class labels: A predefined set of class labels is defined for the dataset. Each object in the image is assigned a class label, indicating the category or type of the object.

4. Bounding box annotations: For each object instance in an image, a bounding box is defined. The bounding box represents the rectangular region enclosing the object. It is specified by the coordinates of the top-left corner, width, and height of the box.

5. Additional metadata: Pascal VOC format allows the inclusion of additional metadata for each image or annotation. This can include information like the source of the image, the author, or any other relevant details. The Edge Impulse uploader currently doesn't import these metadata.

The structure of an annotation file in Pascal VOC format typically follows this pattern:

cubes.23im33f2.xml:

Plain CSV

The Plain CSV format is a very simple format: a CSV annotation file is stored in the same directory as the images. We support both "Single Label" and "Object Detection" labeling methods for this format.

An Plain CSV dataset can follow this directory structure:

Annotation Format:

• Each line in the CSV file represents an object annotation.

• The values in each line are separated by commas.

CSV Columns (Single Label):

The Plain CSV format (single Label) just contains the file_name and the class:

• file_name: The filename of the image.

• classes: The class label or category of the image.

_annotations_single_label.csv example:

CSV Columns (Object Detection):

This Plain CSV format is similar to the TensorFlow Object Detection Dataset format. In this format, the CSV file contains the following columns:

• file_name: The filename of the image.

• classes: The class label or category of the object.

• xmin: The x-coordinate of the top-left corner of the bounding box.

• ymin: The y-coordinate of the top-left corner of the bounding box.

• xmax: The x-coordinate of the bottom-right corner of the bounding box.

• ymax: The y-coordinate of the bottom-right corner of the bounding box.

Each row represents an annotated object in an image. In the following example, there are three objects in cubes_training_0.jpg: a blue, a green and a red cube, two objects in cubes_training_1.jpg, etc... The bounding box coordinates are specified as the top-left corner (xmin, ymin) and the bottom-right corner (xmax, ymax).

_annotations_bounding_boxes.csv example:

YOLO TXT

The YOLO TXT format is a specific text-based annotation format mostly used in conjunction with the YOLO object detection algorithm. This format represents object annotations for an image in a plain text file.

1. File Structure:

   • Each annotation is represented by a separate text file.

   • The text file has the same base name as the corresponding image file.

   • The file extension is .txt.

Example:

1. Annotation Format:

   • Each line in the TXT file represents an object annotation.

   • Each annotation line contains space-separated values representing different attributes.

   • The attributes in each line are ordered as follows: class_label, normalized bounding box coordinates (center_x, center_y, width, height).

2. Class label:

   • The class label represents the object category or class.

   • The class labels are usually represented as integers, starting from 0 or 1.

   • Each class label corresponds to a specific object class defined in the dataset.

3. Normalized Bounding Box Coordinates:

   • The bounding box coordinates represent the location and size of the object in the image.

   • The coordinates are normalized to the range [0, 1], where (0, 0) represents the top-left corner of the image, and (1, 1) represents the bottom-right corner.

   • The normalized bounding box coordinates include the center coordinates (center_x, center_y) of the bounding box and its width and height.

   • The center coordinates (center_x, center_y) are relative to the width and height of the image, where (0, 0) represents the top-left corner, and (1, 1) represents the bottom-right corner.

   • The width and height are also relative to the image size.

Here's an example of a YOLO TXT annotation file format for a single object:

For instance: cubes-23im33f2.txt

Each line represent a given normalized bounding box for the corresponding cubes-23im33f2.jpg image.

1. Mapping the Class Label:

   • The classes.txt, classes.names or data.yaml (used by Roboflow YOLOv5 PyTorch export format) files contain configuration values used by the model to locate images and map class names to class_ids.

For example with the cubes on a conveyor belt dataset with the classes.txt file:

After collecting data for your project, you can now create your Impulse. A complete Impulse will consist of 3 main building blocks: input block, processing block and a learning block.

This view is one of the most important, here you will build your own machine learning pipeline.

Impulse example for movement classification using accelerometer data

Impulse example for object detection using images

Input block

The input block indicates the type of input data you are training your model with. This can be time series (audio, vibration, movements) or images.

Time series (audio, vibration, movements)

• The input axes field lists all the axis referenced from your training dataset

• The window size is the size of the raw features that is used for the training

• The window increase is used to artificially create more features (and feed the learning block with more information)

• The frequency is automatically calculated based on your training samples. You can modify this value but you currently cannot use values lower than 0.000016 (less than 1 sample every 60s).

• Zero-pad data: Adds zero values when raw feature is missing

Below is a sketch to summarize the role of each parameters:

Images

• Axes: Images

• Image width & height: Most of our pre-trained models work with square images.

• Resize mode: You have three options, Squash, Fit to the shortest axis, Fit to the longest axis

Processing blocks

You don't have much experience with DSP? No problem, Edge Impulse usually uses a star to indicate the most recommended processing block based on your input data as shown in the image below.

Learning blocks

Learning blocks available with time-series projects

Learning blocks available with image projects

Learning blocks available with object detection projects

All collected data for each project can be viewed on the Data acquisition tab. You can see how your data has been split for train/test set as well as the data distribution for each class in your dataset. You can also send new sensor data to your project either by file upload, WebUSB, Edge Impulse API, or Edge Impulse CLI.

Add data to your project

Collect data

The panel on the right allows you to collect data directly from any fully supported platform:

When using the Edge Impulse for Linux CLI, run edge-impulse-linux --clean and it will add your platform to the device list of your project. You will then will be able to interact with it from the Collect data panel.

Upload existing datasets

Dataset train/test split ratio

The train/test split is a technique for training and evaluating the performance of a machine learning algorithms. It indicates how your data is split between training and testing samples. For example, an 80/20 split indicates that 80% of the dataset is used for model training purposes while 20% is used for model testing.

This section also shows how your data samples in each class are distributed to prevent imbalanced datasets which might introduce bias during model training.

Data acquisition filter

Manually navigating to some categories of data can be time-consuming, especially when dealing with a large dataset. The data acquisition filter enables the user to filter data samples based on some criteria of choice. This can be based on:

• Label - class to which a sample represents.

• Sample name - unique ID representing a sample.

• Signature validity

• Enabled and disabled samples

• Length of sample - duration of a sample.

The filtered samples can then be manipulated by editing labels, deleting, and moving from the training set to the testing set (and vice versa), a shown in the image above.

Data sample actions

The data manipulations above can also be applied at the data sample level just by simply navigating to the individual data sample then clicking ⋮ and selecting the type of action you might want to perform to the specific sample. This might be renaming , editing its label, disabling, cropping, splitting, downloading, and even deleting the sample when desired.

Cropping samples

To crop a data sample, go to the sample you want to crop and click ⋮, then select Crop sample. You can specific a length, or drag the handles to resize the window, then move the window around to make your selection.

Made a wrong crop? No problem, just click Crop sample again and you can move your selection around. To undo the crop, just set the sample length to a high number, and the whole sample will be selected again.

Splitting data sample

Besides cropping you can also split data automatically. Here you can perform one motion repeatedly, or say a keyword over and over again, and the events are detected and can be stored as individual samples. This makes it easy to very quickly build a high-quality dataset of discrete events. To do so head to Data Acquisition, record some new data, click, and select Split sample. You can set the window length, and all events are automatically detected. If you're splitting audio data you can also listen to events by clicking on the window, the audio player is automatically populated with that specific split.

Samples are automatically centered in the window, which might lead to problems on some models (the neural network could learn a shortcut where data in the middle of the window is always associated with a certain label), so you can select "Shift samples" to automatically move the data a little bit around.

Splitting data is - like cropping data - non-destructive. If you're not happy with a split just click Crop sample and you can move the selection around easily.

Labeling queue

If you are not dealing with an object detection task, you can simply disable the labeling queue bar by going to Dashboard > Project info > Labeling method and clicking the dropdown and selecting "one label per data item" as shown in the image below.