The API references for the ingestion service, remote management service, and the studio API; plus SDK documentation for the acquisition and inferencing libraries can be found in the API references.

Services

• Studio API reference

• Ingestion service

• Remote management service

SDK documentation

• Data acquisition SDK

• Inferencing SDK

• Edge Impulse for Linux SDKs for Node.js, Python, Go and C++

Machine learning (ML) is a way of writing computer programs. Specifically, it’s a way of writing programs that process raw data and turn it into information that is meaningful at an application level.

For example, one ML program might be designed to determine when an industrial machine has broken down based on readings from its various sensors, so that it can alert the operator. Another ML program might take raw audio data from a microphone and determine if a word has been spoken, so it can activate a smart home device.

Unlike normal computer programs, the rules of ML programs are not determined by a developer. Instead, ML uses specialized algorithms to learn rules from data, in a process known as training.

In a traditional piece of software, an engineer designs an algorithm that takes an input, applies various rules, and returns an output. The algorithm’s internal operations are planned out by the engineer and implemented explicitly through lines of code. To predict breakdowns in an industrial machine, the engineer would need to understand which measurements in the data indicate a problem and write code that deliberately checks for them.

This approach works fine for many problems. For example, we know that water boils at 100°C at sea level, so it’s easy to write a program that can predict whether water is boiling based on its current temperature and altitude. But in many cases, it can be difficult to know the exact combination of factors that predicts a given state. To continue with our industrial machine example, there might be various different combinations of production rate, temperature, and vibration level that might indicate a problem but are not immediately obvious from looking at the data.

To create an ML program, an engineer first collects a substantial set of training data. They then feed this data into a special kind of algorithm, and let the algorithm discover the rules. This means that as ML engineers, we can create programs that make predictions based on complex data without having to understand all of the complexity ourselves.

Through the training process, the ML algorithm builds a model of the system based on the data we provide. We run data through this model to make predictions, in a process called inference.

Where can machine learning help?

Machine learning is an excellent tool for solving problems that involve pattern recognition, especially patterns that are complex and might be difficult for a human observer to identify. ML algorithms excel at turning messy, high-bandwidth raw data into usable signals, especially combined with conventional signal processing.

For example, the average person might struggle to recognize the signs of a machine failure given ten different streams of dense, noisy sensor data. However, a machine learning algorithm can often learn to spot the difference.

But ML is not always the best tool for the job. If the rules of a system are well defined and can be easily expressed with hard-coded logic, it’s usually more efficient to work that way.

What is embedded ML?

Recent advances in microprocessor architecture and algorithm design have made it possible to run sophisticated machine learning workloads on even the smallest of microcontrollers. Embedded machine learning, also known as TinyML, is the field of machine learning when applied to embedded systems such as these.

There are some major advantages to deploying ML on embedded devices. The key advantages are neatly expressed in the unfortunate acronym BLERP, coined by Jeff Bier. They are:

Bandwidth—ML algorithms on edge devices can extract meaningful information from data that would otherwise be inaccessible due to bandwidth constraints.

Latency—On-device ML models can respond in real-time to inputs, enabling applications such as autonomous vehicles, which would not be viable if dependent on network latency.

Economics—By processing data on-device, embedded ML systems avoid the costs of transmitting data over a network and processing it in the cloud.

Reliability—Systems controlled by on-device models are inherently more reliable than those which depend on a connection to the cloud.

Privacy—When data is processed on an embedded system and is never transmitted to the cloud, user privacy is protected and there is less chance of abuse.

Learn more

The best way to learn about embedded machine learning is to see it for yourself. To train your own model and deploy it to any device, including your mobile phone, follow our Getting Started guide.

How can I share my Edge Impulse project?

The enterprise version of Edge Impulse offers team collaboration on projects, go to Dashboard, find the Collaborators section, and click the '+' icon. If you have an interesting research or community project we can enable collaboration on the free version of Edge Impulse as well, by emailing hello@edgeimpulse.com.

You can also create a public version of your Edge Impulse project. This makes your project available to the whole world - including your data, your impulse design, your models, and all intermediate information - and can easily be cloned by anyone in the community. To do so, go to Dashboard, and click Make this project public.

What are the minimum hardware requirements to run the Edge Impulse inferencing library on my embedded device?

The minimum hardware requirements for the embedded device depends on the use case, anything from a Cortex-M0+ for vibration analysis to Cortex-M4F for audio, Cortex-M7 for image classification to Cortex-A for object detection in video, view our inference performance metrics for more details.

What frameworks does Edge Impulse use to train the machine learning models?

We use a wide variety of tools, depending on the machine learning model. For neural networks we typically use TensorFlow and Keras, for object detection models we use TensorFlow with Google's Object Detection API, and for 'classic' non-neural network machine learning algorithms we mainly use sklearn. For neural networks you can see (and modify) the Keras code by clicking ⋮, and selecting Switch to expert mode.

Another big part of Edge Impulse are the processing blocks, as they clean up the data, and already extract important features from your data before passing it to a machine learning model. The source code for these processing blocks can be found on GitHub: edgeimpulse/processing-blocks (and you can build your own processing blocks as well).

What engine does Edge Impulse use to compile the Impulse?

It depends on the hardware.

For general-purpose MCUs we typically use EON Compiler with TFLite Micro kernels (including hardware optimization, e.g. via CMSIS-NN, ESP-NN).

On Linux, if you run the Impulse on CPU, we use TensorFlow Lite.

For accelerators we use a wide variety of other runtimes, e.g. hardcoded network in silicon for Syntiant, custom SNN-based inference engine for Brainchip Akida, DRP-AI for Renesas RZV2L, etc...

Is there a downside to enabling the EON Compiler?

The EON Compiler compiles your neural networks to C++ source code, which then gets compiled into your application. This is great if you need the lowest RAM and ROM possible (EON typically uses 30-50% less memory than TensorFlow Lite) but you also lose some flexibility to update your neural networks in the field - as it is now part of your firmware.

By disabling EON we place the full neural network (architecture and weights) into ROM, and load it on demand. This increases memory usage, but you could just update this section of the ROM (or place the neural network in external flash, or on an SD card) to make it easier to update.

Can I use a model that has been trained elsewhere in Edge Impulse?

Yes you can! Check out our documentation on Bringing your own model (BYOM) into your Edge Impulse project, and using the Edge Impulse Python SDK!

How does the feature explorer visualize data that has more that 3 dimensions?

Edge Impulse uses UMAP (a dimensionality reduction algorithm) to project high dimensionality input data into a 3 dimensional space. This even works for extremely high dimensionality data such as images.

Does Edge impulse integrate with other cloud services?

Yes. The enterprise version of Edge Impulse can integrate directly with your cloud service to access and transform data.

What is the typical power consumption of the Edge Impulse machine learning processes on my device?

Simple answer: To get an indication of time per inference we show performance metrics in every DSP and ML block in the Studio. Multiply this by the active power consumption of your MCU to get an indication of power cost per inference.

More complicated answer: It depends. Normal techniques to conserve power still apply to ML, so try to do as little as possible (do you need to classify every second, or can you do it once a minute?), be smart about when to run inference (can there be an external trigger like a motion sensor before you run inference on a camera?), and collect data in a lower power mode (don't run at full speed when sampling low-resolution data, and see if your sensor can use an interrupt to wake your MCU - rather than polling).

Also see Analyse Power Consumption in Embedded ML Solutions.

What is the .eim model format for Edge Impulse for Linux?

See .eim models? on the Edge Impulse for Linux pages.

How is the labeling of the data performed?

Using the Edge Impulse Studio data acquisition tools (like the serial daemon or data forwarder), you can collect data samples manually with a pre-defined label. If you have a dataset that was collected outside of Edge Impulse, you can upload your dataset using the Edge Impulse CLI, data ingestion API, web uploader, enterprise data storage bucket tools or enterprise upload portals. You can then utilize the Edge Impulse Studio to split up your data into labeled chunks, crop your data samples, and more to create high quality machine learning datasets.

Can I use an unsupported development board or a custom PCB (with a different microcontroller or microprocessor) with Edge Impulse?

Yes! A "supported board" simply means that there is an official or community-supported firmware that has been developed specifically for that board that helps you collect data and run impulses. Edge Impulse is designed to be extensible to computers, smartphones, and a nearly endless array of microcontroller build systems.

You can collect data and upload it to Edge Impulse in a variety of ways. For example:

• Transmitting data to the Data forwarder

• Using the Edge Impulse for Linux SDK

• By uploading files directly (e.g. CBOR, JSON, CSV, WAV, JPG, PNG)

Your trained model can be deployed as part a C++ library. It requires some effort, but most build systems will work with our C++ library, as long as that build system has a C++ compiler and there is enough flash/RAM on your device to run the library (which includes the DSP block and model).

After creating your Edge Impulse Studio project, you will be directed to the project's dashboard. The dashboard gives a quick overview of your project such as your project ID, the number of devices connected, the amount of data collected, the preferred labeling method, among other editable properties. You can also enable some additional capabilities to your project such as collaboration, making your project public, and showcasing your public projects using Markdown READMEs as we will see.

The figure below shows the various sections and widgets of the dashboard that we will cover here.

1. Getting Started

The Getting Started section is here to help. You can choose from 3 different options to get started:

• Add existing data: When selecting this option, you can then choose to Upload data from your computer or to Add storage bucket

• Collect new data: When selecting this option, the getting started guide will ask you to either Scan QR code to connect to your phone, Connect to your computer, or to Connect your device or developer board. Make sure that your device or development board is flashed with the Edge Impulse official firmware.

• Upload your model: This option will change the default workflow. See BYOM to learn more about how to import your existing model to Edge Impulse studio.

2. Making your project public

To share your private project with the world, and click Make this project public.

By doing this, all of your data, block configurations, intermediate results, and final models will be shared with the world. Your project will be publicly accessible and can be cloned with a single click with the provided URL:

3. Run this model

When you have a trained model available in your project, this card will appear to let you test your model with your phone or your computer. This is particularly useful to validate the behaviour of your model before integrating it into your targeted embedded firmware.

4. Collaboration

You can invite up to three collaborators to join and contribute to your project. To have unlimited collaborators, your project needs to be part of an organization to access unlimited team collaborations.

To add a collaborator, go to your project's dashboard and find the "Collaborators" widget. Click the '+' icon and type the username or e-mail address of the other user. The user will be invited to create an Edge Impulse account if it doesn't exist.

The user will be automatically added to the project and will get an email notification inviting them to start contributing to your project. To remove a user, simply click on the three dots besides the user then tap ‘Delete’ and they will be automatically removed.

5. Showcasing your public projects with Markdown READMEs

The project README enables you to explain the details of your project in a short way. Using this feature, you can add visualizations such as images, GIFs, code snippets, and text to your project in order to bring your colleagues and project viewers up to speed with the important details of your project. In your README you might want to add things like:

• What the project does

• Why the project is useful

• Motivations of the project

• How to get started with the project

• What sensors and target deployment devices you used

• How you plan to improve your project

• Where users can get help with your project

To create your first README, navigate to the "about this project" widget and click "add README"

For more README inspiration, check out the public Edge Impulse project tutorials below:

• Responding to your voice.

• Continuous motion recognition.

• Adding sight to your sensors.

• Recognize sounds from audio.

6. Project info

The project info widget shows the project's specifications such as the project ID, labeling method, and latency calculations for your target device.

• The project ID is a unique numerical value that identifies your project. Whenever you have any issue with your project on the studio, you can always share your project ID on the forum for assistance from edge Impulse staff.

• On the labeling method dropdown, you need to specify the type of labeling your dataset and model expect. This can be either one label per data item or bounding boxes. Bounding boxes only work for object detection tasks in the studio. Note that if you interchange the labeling methods, learning blocks will appear to be hidden when building your impulse.

• One of the amazing Edge Impulse superpowers is the latency calculation component. This is an approximate time in milliseconds that the trained model and DSP operations are going to take during inference based on the selected target device. This hardware in the loop approach ensures that the target deployment device compute resources are not underutilized or over-utilized. It also saves developers' time associated with numerous inference iterations back and forth the studio in search of optimum models.

7. Block Outputs

In the Block Output section, you can download the results of the DSP and ML operations of your impulse.

The downloadable assets include the extracted features, Tensorflow SavedModel, and both quantized and unquantized TensorFlow lite models. This is particularly helpful when you want to perform other operations to the output blocks outside the Edge Impulse studio. For example, if you need a TensorflowJS model, you will just need to download the TensorFlow saved model from the dashboard and convert it to TensorFlowJS model format to be served on a browser.

8. Performance Settings

This section consists of editable parameters that directly affect the performance of the studio when building your impulse. Depending on the selected or available settings, your jobs can either be fast or slow.

The use of GPU for training and Parallel DSP jobs is currently an internal experimental feature that will be soon released.

9. Administrative Zone

To bring even more flexibility to projects, the administrative zone gives developers the power to enable other additional features that are not found in edge impulse projects by default. Most of these features are usually advanced features intended for organizations or sometimes experimental features.

To activate these features you just need to check the boxes against the specific features you want to use and click save experiments.

10. Danger Zone

The danger zone widget consists of irrevocable actions that let you to:

• Perform train/test split. This action re-balances your dataset by splitting all your data automatically between the training and testing set and resets the categories for all data.

• Launch the getting started wizard. This will remove all data, and clear out your impulse.

• Transfer ownership. This action is available for users who have one or more organizations linked with their accounts. With it, you can start working on a project with your user profile and then transfer the ownership to your organization.

• Delete your project. This action removes all devices, data, and impulses from your project.

• Delete all data in this project.

You can upload an already existing dataset to your project directly through the Edge impulse Studio. The data should be in the Data Acquisition format (CBOR, JSON, CSV), or as WAV, JPG or PNG files.

To upload data using the uploader, go to the Data acquisition page and click on the uploader button as shown in the image below:

When uploading your data, you can choose the category you want your data to fall in i.e training set, testing set or automatically split the dataset between training and testing set. You can also choose whether to infer labels from files name or enter a label of which the files should automatically fall in.

There is a wide variety of devices that you can connect to your Edge Impulse project. These devices can help you collect datasets for your project, test your trained ML model and even deploy your ML model directly to your development board with a pre-built binary application (for fully supported development platforms).

On the Devices tab, you'll find a list of all your connected devices and a guide on how to connect new devices that are currently supported by Edge Impulse.

To connect a new device, click on the Connect a new device button on the top right of your screen.

You will get a pop-up with multiple options of devices you can connect to your Edge Impulse project. Available options include:

• Connecting a fully supported development board.

• Using your mobile phone.

• Using your computer (Linux/Mac).

• Using the serial port to connect devices that are currently not fully supported.

Welcome to Edge Impulse! We enable developers to create the next generation of intelligent device solutions with . In the documentation you'll find user guides, tutorials and API documentation. For support, visit the .

Get started with any device

Follow these three steps to build your first embedded Machine Learning model - no worries, you can use almost any device to get started.

1. You'll need some data:

   • If you have an existing development board or device, you can collect data with a few lines of code using the or the SDK.

   • If you want to collect live data from a supported development kit, select your board from the list of and follow the instructions to connect your board to edge impulse.

   • If you already have a dataset, you can upload it via the .

   • If you have a mobile phone you can use it as a sensor to collect data, see .

2. Try the tutorials on , , , or . These will let you build machine learning models that detect things in your home or office.

3. After training your model you can run your model on your device:

   • If you want to integrate the model with your own firmware or project you can export your complete model (including all signal processing code and machine learning models) to a C++ or Arduino library with no external dependencies (open source and royalty-free), see .

   • If you have a fully supported development board (or your mobile phone) you can build new firmware - which includes your model - directly from the UI. It doesn't get easier than that!

   • If you have a gateway, a computer or a web browser where you want to run your model, you can export to and run it anywhere you can run JavaScript.

Suitable for any type of embedded ML application

We have some great tutorials, but you have full freedom in the models that you design in Edge Impulse. You can plug in new signal processing blocks, and completely new neural networks. See and .

API Documentation

Enterprise version

All collected data for each project can be viewed on the Data acquisition tab. You can see how your data has been split for train/test set as well as the data distribution for each class in your dataset. You can also send new sensor data to your project either by file upload, WebUSB, Edge Impulse API, or Edge Impulse CLI.

Add data to your project

Record new data

The panel on the right allows you to collect data directly from any fully supported platform:

When using the Edge Impulse for Linux CLI, run edge-impulse-linux --clean and it will add your platform to the device list of your project. You will then will be able to interact with it from the Record new data panel.

Other methods

Dataset train/test split ratio

The train/test split is a technique for training and evaluating the performance of a machine learning algorithms. It indicates how your data is split between training and testing samples. For example, an 80/20 split indicates that 80% of the dataset is used for model training purposes while 20% is used for model testing.

This section also shows how your data samples in each class are distributed to prevent imbalanced datasets which might introduce bias during model training.

Data acquisition filter

Manually navigating to some categories of data can be time consuming, especially when dealing with a large dataset. The data acquisition filter enables the user to filter data samples based on some criteria of choice. This can be based on:

• Label - class to which a sample represents.

• Sample name - unique ID representing a sample.

• Signature validity

• Enabled and disabled samples

• Length of sample - duration of a sample.

The filtered samples can then be manipulated by editing label, deleting, moving from trains set to test set and vise versa a shown in the image above.

The data manipulations above can also be applied at the data sample level just by simply navigating to the individual data sample then clicking ⋮ and selecting the type of action you might want to perform to the specific sample. This might be renaming , editing its label, disabling, cropping, splitting, downloading, and even deleting the sample when desired.

Cropping samples

To crop a data sample, go to the sample you want to crop and click ⋮, then select Crop sample. You can specific a length, or drag the handles to resize the window, then move the window around to make your selection.

Made a wrong crop? No problem, just click Crop sample again and you can move your selection around. To undo the crop, just set the sample length to a high number, and the whole sample will be selected again.

Splitting data sample

Besides cropping you can also split data automatically. Here you can perform one motion repeatedly, or say a keyword over and over again, and the events are detected and can be stored as individual samples. This makes it easy to very quickly build a high-quality dataset of discrete events. To do so head to Data acquisition, record some new data, click, and select Split sample. You can set the window length, and all events are automatically detected. If you're splitting audio data you can also listen to events by clicking on the window, the audio player is automatically populated with that specific split.

Samples are automatically centered in the window, which might lead to problems on some models (the neural network could learn a shortcut where data in the middle of the window is always associated with a certain label), so you can select "Shift samples" to automatically move the data a little bit around.

Splitting data is - like cropping data - non-destructive. If you're not happy with a split just click Crop sample and you can move the selection around easily.

Labelling queue

The labelling queue will only appear on your data acquisition page if you are dealing with an object detection tasks. The labelling queue shows a list of images that have been staged for annotation for your project.

If you are not dealing with an object detection task, you can simply disable the labelling queue bar by going to Dashboard > Project info > Labeling method and clicking the dropdown and selecting "one label per data item" as shown in the image below.

The CSV Wizard allows users with larger or more complex datasets to easily upload their data without having to worry about converting it to the .

To access the CSV Wizard, navigate to the Data Acquisition tab of your Edge Impulse project and click on the CSV Wizard button:

How to use the CSV Wizard

We can take a look at some sample data from a Heart Rate Monitor (Polar H10). We can see there is a lot of extra information we don’t need:

Step 1: Upload a file

Choose a CSV file to upload and select "Upload File". The file will be automatically analyzed and the results will be displayed in the next step. Here I have selected an export from a HR monitor. You can try it out yourself by downloading this file:

Step 2: Analyze your data

When processing your data, we will check for the following:

• Does this data contain a label?

• Is this data time series data?

• Is this data raw sensor data or processed features?

• Is this data separated by a standard delimiter?

• Is this data separated by a non-standard delimiter?

If there are settings that need to be adjusted, (for the start of your data you can select skip first x lines or no header, and adjust the delimiter) you can do so before selecting looks good, next"**.

Step 3: About your data

Here you can select the timestamp column, or row and the frequency of the timestamps. If you do not have a timestamp column, you can select No timestamp column and add a timestamp later. If you do have a timestamp column you can select: the timestamp format, e.g. full timestamp, and the frequency of the timestamps, overriding is also possible via Override timestamp difference. For example Selecting 20000 will give you the detected frequency of: 0.05 Hz.

Step 4: CSV Wizard: About your values

Here you can select the label column, or row. If you do not have a label column, you can select No (no worries, you can provide this when you upload data) and add a label later. If you do have a label column you can select: Yes it's "Value" The CSV Wizard allows users with larger or more complex datasets to easily upload their data without having to worry about converting it to CBOR format. You can also select the columns that contain your values.

Step 5: Split up your samples

How long do you want your samples to be?

In this section, you can set a length limit to your sample size. For example, if your CSV contains 30 seconds of data, when setting a limit of 3000ms, it will create 10 distinct data samples of 3 seconds.

Congratulations! 🚀 You have successfully created a CSV transform with the CSV Wizard. You can now save this transform and use it to process your data.

What happens next?

Any CSV files that you upload into your project - whether it's through the uploader, the CLI, the API or through data sources - will now be processed according to the rules you set up with the CSV Wizard!

Congratulations, you've trained your first embedded machine learning model! This page lists next steps you can take to make your devices smarter.

Run your model on a real device

You've ran your model in the browser, but you can also run it on a wide variety of devices. Head to the development boards section for a full overview. If you have a device that is not supported, no problem, you can export your model as a C++ library that runs on any embedded device. See Running your impulse locally for more information.

More than audio

Making a machine learning model that responds to your voice is cool, but you can do a lot more with Edge Impulse. Here are a number of tutorials to get you started:

• Continuous motion recognition

• Recognize sounds from audio

• Responding to your voice

• Adding sight to your sensors

• Object detection with bounding boxes

• Object detection with centroids

• Sensor fusion

Make your model more robust by adding more data

Your model was trained on +/- 20 seconds of data, which is a very small amount of data. To make your model more robust you can add more data.

• If your model does not respond well enough on your keyword (e.g. if you have someone saying the word in a different tone or pitch), record some more data of the keyword.

• If the model is too sensitive (triggers when you say something else), then say some different words and label them with the 'unknown' class.

You can record new data from your computer, your phone, or a development board. Go to Data acquisition and click Show options for instructions. Then, to split your data into individual samples, click the three dots next to a sample, and select Split sample (more info).

Share your project with the world

Think your model is awesome, and want to share it with the world? Go to Dashboard and click Make this project public. This will make your whole project - including all data, machine learning models and visualizations - available, and can be viewed and cloned by anyone with the URL.

More questions? Ask us on the forums!

Do you have any other questions or want to share your awesome ideas? Head to the forum!

In object detection ML projects, labeling is the process of defining regions of interest in the frame.

Manually labeling images can become tedious and time-consuming, especially when dealing with huge datasets. This is why Edge Impulse studio provides an AI-assisted labeling tool to help you in your labeling workflows.

To use the labeling queue, you will need to set your Edge Impulse project as an "object detection" project. The labeling queue will only display the images that have not been labeled.

Currently, it only works to define bounding boxes (ingestion format used to train both MobileNetv2 SSD and FOMO models).

AI-Assisted labeling

There are 3 ways you can use to perform AI-assisted labeling on the Edge Impulse Studio:

• Using yolov5

• Using your own model

• Using object tracking

Using YOLOv5

By utilizing an existing library of pre-trained object detection models from YOLOv5 (trained with the COCO dataset), common objects in your images can quickly be identified and labeled in seconds without needing to write any code!

To label your objects with YOLOv5 classification, click the Label suggestions dropdown and select “Classify using YOLOv5.” If your object is more specific than what is auto-labeled by YOLOv5, e.g. “coffee” instead of the generic “cup” class, you can modify the auto-labels to the left of your image. These modifications will automatically apply to future images in your labeling queue.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes!

Using your own model

You can also use your own trained model to predict and label your new images. From an existing (trained) Edge Impulse object detection project, upload new unlabeled images from the Data Acquisition tab.

From the “Labeling queue”, click the Label suggestions dropdown and select “Classify using ”:

You can also upload a few samples to a new object detection project, train a model, then upload more samples to the Data Acquisition tab and use the AI-Assisted Labeling feature for the rest of your dataset. Classifying using your own trained model is especially useful for objects that are not in YOLOv5, such as industrial objects, etc.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes using your own pre-trained model!

Using Object tracking

If you have objects that are a similar size or common between images, you can also track your objects between frames within the Edge Impulse Labeling Queue, reducing the amount of time needed to re-label and re-draw bounding boxes over your entire dataset.

Draw your bounding boxes and label your images, then, after clicking Save labels, the objects will be tracked from frame to frame:

Now that your object detection project contains a fully labeled dataset, learn how to train and deploy your model to your edge device: check out our tutorial!

We are excited to see what you build with the AI-Assisted Labeling feature in Edge Impulse, please post your project on our forum or tag us on social media, @Edge Impulse!

The Raw Data block generates windows from data samples without any specific signal processing. It is great for signals that have already been pre-processed and if you just need to feed your data into the Neural Network block.

GitHub repository containing all DSP block code: .

Raw data parameters

Scaling

• Scale axes: Multiplies each axis by this number. This can be used to normalize your data between 0 and 1.

How does the raw data block work?

The Raw Data block retrieves raw samples and applies the Scaling parameter.

You can add arbitrary metadata to data items. You can use this for example to track on which site data was collected, where data was imported from, or where the machine that generated the data was placed. Some key use cases for metadata are:

1. Prevent leaking data between your train and validation set. See: below.

2. Synchronisation actions in , for example to remove data in a project if the source data was deleted in the cloud.

3. Get a better understanding of real-world accuracy by seeing how well your model performs when grouped by a metadata key. E.g. whether data on site A performs better than site B.

Viewing and editing metadata in the Studio

Metadata is shown on Data acquisition when you click on a data item. From here you can add, edit and remove metadata keys.

Adding metadata when adding data

It's pretty unpractical to manually add metadata to each data item, so the easiest way is to add metadata when you upload data. You can do this either by:

2. Setting the x-metadata header to a JSON string when calling the ingestion service:

Reading and writing metadata through the API

Using metadata to control your train/validation split

When training an ML model we split your data into a train and a validation set. This is done so that during training you can evaluate whether your model works on data that it has seen before (train set) and on data that it has never seen before (validation set) - ideally your model performs similarly well on both data sets: a sign that your model will perform well in the field on completely novel data.

However, this can give a false sense of security if data that is very similar ends up in both your train and validation set ("data leakage"). For example:

• You split a video into individual frames. These images don't differ much from frame to frame; and you don't want some frames in the train, and some in the validation set.

• You're building a sleep staging algorithm, and look at 30 second windows. From window to window the data for one person will look similar, so you don't want one window in the train, another in the validation set for the same person in the same night.

By default we split your training data randomly in a train and validation set (80/20 split) - which does not prevent data leakage, but if you tag your data items with metadata you can avoid this. To do so:

1. Tag all your data items with metadata.

2. Go to any ML block and under Advanced training settings set 'Split train/validation set on metadata key' to a metadata key (f.e. video_file).

Now every data item with the same metadata value for video_file will always be grouped together in either the train or the validation set; so no more data leakage.

The data sources page is actually much more than just adding data from external sources. It let you create complete automated data pipelines so you can work on your active learning strategies.

From there, you can import datasets from existing cloud storage buckets, automate and schedule the imports, and, trigger actions such as explore and label your new data, retrain your model, automatically build a new deployment task and more.

Add a data source

Click in + Add new data source and select where your data lives:

You can either use:

• AWS S3 buckets

• Google Cloud Storage

• Any S3-compatible bucket

• Don't import data (if you just need to create a pipeline)

Click on Next, provide credentials:

Click on Verify credentials:

Here, you have several options to automatically label your data:

Infer from folder name

In the example above, the structure of the folder is the following:

The labels will be picked from the folder name and will be split between your training and testing set using the following ratio 80/20.

Alternatively, you can also organize your folder using the following structure to automatically split your dataset between training and testing sets:

Infer from file name

When using this option, only the file name is taken into account. The part before the first . will be used to set the label. E.g. cars.01741.jpg will set the label to cars.

Keep the data unlabeled

All the data samples will be unlabeled, you will need to label them manually before using them.

Finally, click on Next, post-sync actions.

From this view, you can automate several actions:

• Recreate data explorer
• Retrain model

  If needed, will retrain your model with the same impulse. If you enable this you'll also get an email with the new validation and test set accuracy.

  Note: You will need to have trained your project at least once.

• Create new version

  Store all data, configuration, intermediate results and final models.

• Create new deployment

  Builds a new library or binary with your updated model. Requires 'Retrain model' to also be enabled.

Run the pipeline

Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.

Run the pipeline from the UI

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.

Run the pipeline from code

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.

Schedule your pipeline jobs

By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.

Once the pipeline has successfully finish, you will receive an email like the following:

Webhooks

Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:

Edit your pipeline

As of today, if you want to update your pipeline, you need to edit the configuration json available in ⋮ -> Run pipeline from code.

Here is an example of what you can get if all the actions have been selected:

Free projects have only access to the above builtinTransformationBlock.

Select Copy as pipeline step and paste it to the configuration json file.

The Image block is dedicated to computer vision applications. It normalizes image data, and optionally reduce the color depth.

GitHub repository containing all DSP block code: .

Image parameters

• Color depth: Color depth to use (RGB or grayscale)

How does the image block work?

The Image performs normalization, converting each pixel's channel of the image to a float value between 0 and 1. If Grayscale is selected, each pixel is converted to a single value following the (Y' component only).

The Spectrogram processing block extracts time and frequency features from a signal. It performs well on audio data for non-voice recognition use cases, or on any sensor data with continuous frequencies.

GitHub repository containing all DSP block code: .

Spectrogram parameters

Spectrogram

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• FFT size: The size of the FFT for each frame. Will zero pad or clip if frame length in samples does not equal FFT size.

Normalization

• Noise floor (dB): signal lower than this level will be dropped

How does the spectrogram block work?

It first divides the window in multiple overlapping frames. The size and number of frames can be adjusted with the parameters Frame length and Frame stride. For example with a window of 1 second, frame length of 0.02s and stride of 0.01s, it will create 99 time frames.

An FFT is then calculated for each frame. The number of frequency features for each frame is equal to the FFT size parameter divided by 2 plus 1. We recommend keeping the FFT size a power of 2 for performances purpose. Finally the Noise floor value is applied to the power spectrum.

The features generated by the Spectrogram block are equal to the number of generated time frames times the number of frequency features.

Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio):

• Raw Data

• Flatten

• Image

• Spectral features

• Spectrogram

• Audio MFE

• Audio MFCC

• Audio Syntiant

• IMU Syntiant

The source code of these blocks are available in the Edge Impulse processing blocks GitHub repository.

If you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing, follow our tutorial on Building custom processing blocks.

The IMU Syntiant block rescales raw data to 8 bits values to match the NDP101/120 chip input requirements.

Parameters

Scaling

• Scale 16 bits to 8 bits: Scale data to 8-bits values in the [-1, 1] range, raw data is divided by 2G (2 * 9.80665). Using Edge Impulse official firmwares, this parameter should be enabled as raw data is not rescaled. If this parameter is disabled the data samples will not be rescaled, you should disable this parameter if your raw data samples are already normalized to the [-1, 1] range.

How does the IMU Syntiant block work?

The IMU Syntiant block retrieves raw samples and applies the Scale 16 bits to 8 bits parameter.

Building custom processing blocks is available for everyone but has to be self-hosted. If you want to host it on Edge Impulse infrastructures, you can do that within your organization interface.

In this tutorial, you'll learn how to use Edge Impulse CLI to push your custom DSP block to your organisation and how to make this processing block available in the Studio for all users in the organization.

The Custom Processing block we are using for this tutorial can be found here: https://github.com/edgeimpulse/edge-detection-processing-block. It is written in Python. Please note that one of the beauties with custom blocks is that you can write them in any language as we will host a Docker container and we are not tied to a specific runtime.

Prerequisites

You'll need:

• The Edge Impulse CLI. If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

• Docker desktop installed on your machine. Custom blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• A Custom Processing block running with Docker.

Init and upload your custom DSP block

Inside your Custom DSP block folder, run the following command:

edge-impulse-blocks init --clean

The output will look like this:

? What is your user name or e-mail address (edgeimpulse.com)? 
? What is your password? [hidden]
Edge Impulse Blocks v1.14.3
Attaching block to organization 'Demo Team'

? Choose a type of block 
  Transformation block 
  Deployment block 
❯ DSP block 
  Transfer learning block 

? Enter the name of your block Edge: Detection

? Enter the description of your block: Edge Detection processing block using Canny filters in images

Creating block with config: {
  version: 1,
  config: {
    'edgeimpulse.com': {
      name: 'Edge Detection',
      type: 'dsp',
      description: 'Edge Detection processing block using Canny filters in images',
      organizationId: XXX,
      operatesOn: undefined,
      tlObjectDetectionLastLayer: undefined,
      tlOperatesOn: undefined
    }
  }
}
Your new block 'Edge Detection' has been created in '<PATH>'.
When you have finished building your dsp block, run 'edge-impulse-blocks push' to update the block in Edge Impulse.

Modify or update your custom code if needed and run the following command:

edge-impulse-blocks push            

The output will look similar to this:

Edge Impulse Blocks v1.14.3
? What port is your block listening on? 4446

Archiving 'edge-detection-processing-block'...
Archiving 'edge-detection-processing-block' OK (476 KB) /var/folders/7f/pfcmh61s3hg9c59qd0dkkw5w0000gn/T/ei-dsp-block-c729b4a3ff761b64629617c869e9d934.tar.gz

Uploading block 'Edge Detection' to organization 'Demo Team'...
Uploading block 'Edge Detection' to organization 'Demo Team' OK

Building dsp block 'Edge Detection'...
Job started
...
Building dsp block 'Edge Detection' OK

That's it, now your custom DSP block is hosted on your organization. To make sure it is up and running, in your organisation, go to Custom blocks->DSP and you will see the following screen:

Use your custom hosted DSP block in your projects

To use your DSP block, simply add it as a processing block in the Create impulse view:

Other resources

• Full instruction on how to build processing blocks: Building custom processing blocks

• Blog post: Utilize Custom Processing Blocks in Your Image ML Pipelines

Troubleshooting

Deploy block types are hidden

When running edge-impulse-blocks init for hosting a custom DSP block, ensure you log into an Edge Impulse account that is a member of an Organization. If you are logged into a personal account, you will be presented with the following CLI output:

$ edge-impulse-blocks init
Edge Impulse Blocks v1.16.0
? What is your user name or e-mail address (edgeimpulse.com)? jplunkett@utexas.edu
? What is your password? [hidden]
[CFG] Creating developer profile...
[CFG] Creating developer profile OK
Attaching block to organization 'Jenny Plunkett'
? Choose a type of block (transform, DSP and deploy block types are hidden because you are pushing to a personal profile)
❯ Machine learning block

The Audio MFCC blocks extracts coefficients from an audio signal. Similarly to the Audio MFE block, it uses a non-linear scale called Mel-scale. It is the reference block for speech recognition and can also performs well on some non-human voice use cases.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Audio MFCC parameters

Mel Frequency Cepstral Coefficients

• Number of coefficients: Number of cepstral coefficients to keep after applying Discrete Cosine Transform

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number: The number of triangular filters applied to the spectrogram

• FFT length: The FFT size

• Low frequency: Lowest band edge of Mel-scale filterbanks

• High frequency: Highest band edge of Mel-scale filterbanks

• Window size: The size of sliding window for local cepstral mean normalization. Windows size must be odd.

Pre-emphasis

• Coefficient: The pre-emphasizing coefficient to apply to the input signal (0 equals to no filtering)

• Note: Shift has been removed and set to 1 for all future projects. Older & existing projects can still change this value or use an existing value.

How does the MFCC block work?

The features' extractions adds one extra step to the MFE block resulting in a compressed representation of the filterbanks. A Discrete Cosine Transform is applied on each filterbank to extract cepstral coefficients. 13 coefficients are usually retained, the rest are discarded as they represent fast changes not useful for speech recognition.

The Audio Syntiant processing block extracts time and frequency features from a signal. It is similar to the Audio MFE but performs additional processing specific to the Syntiant NDP101/120 chip. This block can be used only with Syntiant targets.

Audio Syntiant parameters

Log Mel-filterbank energy features

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number (fixed): The number of triangular filters applied to the spectrogram

• FFT length (fixed): The FFT size

• Low frequency (fixed): Lowest band edge of Mel-scale filterbanks

• High frequency (fixed): Highest band edge of Mel-scale filterbanks

Preemphasis

• Coefficient: Pre-emphasis coefficient

Chip

• Features extractor: Syntiant method to generate features, choose accordingly to your chip

How does the Syntiant block work?

The features' extractions is a proprietary algorithm from Syntiant. However parameters are very close to the Audio MFE. Pre-emphasis coefficient is applied first to amplify higher frequencies. The signal is then divided in overlapping frames, defined by the Frame length and Frame stride to extract speech features.

Similarly to the Spectrogram block, the Audio MFE processing block extracts time and frequency features from a signal. However it uses a non-linear scale in the frequency domain, called Mel-scale. It performs well on audio data, mostly for non-voice recognition use cases when sounds to be classified can be distinguished by human ear.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Audio MFE parameters

Mel-filterbank energy features

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number: The number of triangular filters applied to the spectrogram

• FFT length: The FFT size

• Low frequency: Lowest band edge of Mel-scale filterbanks

• High frequency: Highest band edge of Mel-scale filterbanks

Normalization

• Noise floor (dB): signal lower than this level will be dropped

How does the MFE block work?

The features' extractions is similar to the Spectrogram (Frame length, Frame stride, and FFT length parameters are the same) but it adds 2 extra steps.

After computing the spectrogram, triangular filters are applied on a Mel-scale to extract frequency bands. They are configured with parameters Filter number, Low frequency and High frequency to select the frequency band and the number of frequency features to be extracted. The Mel-scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. The idea is to extract more features (more filter banks) in the lower frequencies, and less in the high frequencies, thus it performs well on sounds that can be distinguished by human ear.

The last step is to perform a local mean normalization of the signal, applying the Noise floor value to the power spectrum.

Bring your own model or BYOM allows you to optimize and deploy your own pretrained model (TensorFlow SavedModel, ONNX, or TensorFlow Lite) to any edge device, directly from your Edge Impulse project.

Getting Started

First, create a new project in Edge Impulse.

Also make sure you have your own pretrained model available locally on your computer, in one of the following formats: TensorFlow SavedModel (saved_model.zip), ONNX model (.onnx) or TensorFlow Lite model (.tflite)

For this guide, we will be uploading a pretrained image classification TFLite model for plant disease classification, downloaded from the TensorFlow Dev Hub.

Then, from the Dashboard of your Edge Impulse project under "Getting started", select Upload your model:

Step 1: Upload your model

1. Upload your trained model: Upload a TensorFlow SavedModel (saved_model.zip), ONNX model (.onnx) or TensorFlow Lite model (.tflite) to get started.

2. Model performance: Do you want performance characteristics (latency, RAM and ROM) for a specific device? Select "No" to show the performance for a range of device types, or "Yes" to run performance profiling for any of our available officially supported Edge Impulse development platforms.

After configuring the settings for uploading your model, select Upload your model and wait for your model to upload, you can check the upload status via the "Upload progress" section.

ONNX models

When selecting an ONNX model, you can also upload a .npy file to Upload representative features (Optional). If you upload a set of representative features - for example, your validation set - as an .npy file we can automatically quantize this model for better on-device performance.

Step 2: Process your model

Depending on the model you have uploaded in Step 1, the configuration settings available for Step 2 will change.

For this guide, we have selected the following configuration model settings for optimal processing for an image classification model with input shape (300, 300, 3) in RGB format, Classification model output and 16 output labels: Tomato Healthy, Tomato Septoria Leaf Spot, Tomato Bacterial Spot, Tomato Blight, Cabbage Healthy, Tomato Spider Mite, Tomato Leaf Mold, Tomato_Yellow Leaf Curl Virus, Soy_Frogeye_Leaf_Spot, Soy_Downy_Mildew, Maize_Ravi_Corn_Rust, Maize_Healthy, Maize_Grey_Leaf_Spot, Maize_Lethal_Necrosis, Soy_Healthy, Cabbage Black Rot

After configuring your model settings, select Save model to view your model's on-device performance information for both MCUs and microprocessors (if applicable, depending on your model's arena size).

Check model behavior

Optionally upload test data to ensure correct model settings and proper model processing:

Transfer learning is the process of taking features learned from one problem and leveraging it on a new but related problem. Most of the time these features are learned from large scale datasets with common objects hence making it faster & more accurate to tune and adapt to new tasks. With Edge Impulse's transfer learning block for audio keyword spotting, we take the same transfer learning technique classically used for image classification and apply it to audio data. This allows you to fine-tune a pre-trained keyword spotting model on your data and achieve even better performance than using a classification block, even with a relatively small keyword dataset.

Excited? Train your first keyword spotting model in under 5 minutes with the getting started wizard!

To choose transfer learning as your learning block, go to create impulse and click on Add a Learning Block, and select Transfer Learning (Keyword Spotting).

To choose your preferred pre-trained network, select the Transfer learning tab on the left side of your screen and click choose a different model. A pop up will appear on your screen with a list of models to choose from as shown in the image below.

Edge Impulse uses state of the art MobileNetV1 & V2 architectures trained on an ImageNet dataset as it's pre-trained network for you to fine-tune for your specific application.

Neural Network Settings

Before you start training your model, you need to set the following neural network configurations:

• Number of training cycles: Each time the training algorithm makes one complete pass through all of the training data with back-propagation and updates the model's parameters as it goes, it is known as an epoch or training cycle.

• Learning rate: The learning rate controls how much the models internal parameters are updated during each step of the training process. Or you can also see it as how fast the neural network will learn. If the network overfits quickly, you can reduce the learning rate

• Validation set size: The percentage of your training set held apart for validation, a good default is 20%.

You might also need to enable auto balance to prevent model bias or even enable data augmentation to increase the size of your dataset and have more diverse dataset to prevent overfitting.

The preset configurations just don't work for your model? No worries, Expert Mode is for you! Expert Mode gives you full control of your model so that you can configure it however you want. To enable the expert mode, just click on the "⋮" button and toggle the expert mode.

You can use the expert mode to change your loss function, optimizer, print your model architecture and even set an early stopping callback to prevent overfitting your model.

After extracting meaningful features from the raw signal using signal processing, you can now train your model using a learning block. We provide a number of pre-defined learning blocks:

• .

• .

• .

• .

• .

• .

• .

Miss an architecture? You can , with PyTorch, Keras or scikit-learn.

For most of the learning blocks (except K-means Anomaly Detection), you can use the Switch to expert mode button to access the full Keras API for custom architectures, , and more.

Want to use a novel ML architecture, or load your own transfer learning models into Edge Impulse? Create a custom learning block! It's easy to bring in any training pipeline into the Studio, as long as you can output TFLite or ONNX files. We have end-to-end examples of doing this in Keras, PyTorch and scikit-learn.

This page describes the input and output formats if you want to bring your own model, but a good way to start building a custom learning block is by modifying one of the following example repositories:

• - wraps the Ultralytics YOLOv5 repository (trained with PyTorch) to train a custom transfer learning model.

• - a Keras implementation of transfer learning with EfficientNet B0.

• - a basic multi-layer perceptron in Keras and TensorFlow.

• - a basic multi-layer perceptron in PyTorch.

• - trains a logistic regression model using scikit-learn, then outputs a TFLite file for inferencing using .

Editing built-in blocks

Any built-in block in the Edge Impulse Studio (e.g. classifiers, regression models or FOMO blocks) can be edited locally, and then pushed back as a custom block. This is great if you want to make heavy modifications to these training pipelines, for example to do custom data augmentation. To download a block, go to any ML block in your project, click the three dots, select Edit block locally, and follow the instructions in the README.

Dockerfiles

Training pipelines in Edge Impulse are built on top of Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. To train your own model you'll need to wrap all the required packages, your scripts, and (if you use transfer learning) your pre-trained weights into this container. When running in Edge Impulse the container does not have network access, so make sure you don't download dependencies while running (fine when building the container).

A typical Dockerfile might look like (see the example repositories for more information):

Important: ENTRYPOINT

It's important to create an ENTRYPOINT at the end of the Dockerfile to specify which file to run.

GPU Support

If you want to have GPU support (only for enterprise customers), you'll need cuda packages installed. If you export a learn block from the Studio these will already have the right base packages, so use that Dockerfile as a starting point.

Arguments to your script

The entrypoint (see above in the Dockerfile) will be called with these four parameters:

• --data-directory - where you can find the data (see below for the input/output formats).

• --epochs - number of epochs to train for (set by the user in the UI).

• --learning-rate - learning rate to train with (set by the user in the UI).

• --out-directory - where to write the TFLite or ONNX files (see below for the input/output formats).

Input format

The data directory contains your dataset, after running any DSP blocks, and already split in a train/validation set:

• X_split_train.npy

• Y_split_train.npy

• X_split_test.npy

• Y_split_train.npy

The X_*.npy files are float32 Numpy arrays, already in the right shape (e.g. if you're training on 96x96 RGB images this will be of shape (n, 96, 96, 3)). You can typically load these without any modification into your training pipeline (see the notes after this section for caveats).

The Y_*.npy files are either:

1. int32 Numpy arrays, with four columns (label_index, sample_id, sample_slice_start_ms, sample_slice_end_ms).

2. A JSON array in the form of: [{ "sampleId": 234731, "boundingBoxes": [{ "label": 1, "x": 260, "y": 313, "w": 234, "h": 261 }] } ]

2) is sent if your dataset has bounding boxes, in all other cases 1) is sent.

This regenerates features (if necessary) and then downloads the updated dataset.

Notes on input format for vision models

The input features for vision models are a 3D vector of shape (WIDTH, HEIGHT, CHANNELS), where the channel data is in RGB format and each pixel is scaled 0..1.

If the input to your model is different (e.g. BGR, or scaled 0..255) you'll need to transform the input. This needs to happen as part of your neural network, as the input will always be as stated above. Here's how you can do that:

• If you have a model that requires the input to be scaled 0..255 (e.g. EfficientNet) you can inject a Mul layer that multiplies the input by 255 before passing it to the first hidden layer of your network.
• If you have a model that requires BGR input, rather than RGB input (e.g. Resnet50) you'll need to transpose the first and last channels.

Note on required shape for image models (NCHW vs. NHWC)

Output format

The training pipeline can output either TFLite or ONNX files:

If you output TFLite files

• model.tflite - a TFLite file with float32 inputs and outputs.

• model_quantized_int8_io.tflite - a quantized TFLite file with int8 inputs and outputs.

• saved_model.zip - a TensorFlow saved model (optional).

At least one of the TFLite files is required.

If you output ONNX files

• model.onnx - An ONNX file with float16 or float32 inputs and outputs.

We automatically convert this file to both unquantized and quantized TFLite files after training.

I'm using scikit-learn, I don't have TFLite or ONNX files...

Hosting your custom block

To edit the block, go to:

• Enterprise: go to your organization, Custom blocks > Machine learning.

• Developers: click on your photo on the top right corner, select Custom blocks > Machine learning.

The block is now available from inside any of your Edge Impulse projects. Depending on the data your block operates on, you can add it via:

• Object Detection: Create impulse > Add learning block > Object Detection (Images), then select the block via 'Choose a different model' on the 'Object detection' page.

• Image classification: Create impulse > Add learning block > Transfer learning (Images), then select the block via 'Choose a different model' on the 'Transfer learning' page.

• Audio classification: Create impulse > Add learning block > Transfer Learning (Keyword Spotting), then select the block via 'Choose a different model' on the 'Transfer learning' page.

• Other (classification): Create impulse > Add learning block > Custom classification, then select the block via 'Choose a different model' on the 'Machine learning' page.

• Other (regression): Create impulse > Add learning block > Custom regression, then select the block via 'Choose a different model' on the 'Regression' page.

Object detection output layers

Unfortunately object detection models typically don't have a standard way to go from neural network output layer to bounding boxes. Currently we support the following types of output layers:

• MobileNet SSD

• Edge Impulse FOMO

• YOLOv5 (compatible with Ultralytics YOLOv5 v6)

• YOLOv5 for Renesas DRP-AI

• YOLOX

If you have an object detection model with a different output layer then please contact your user success engineer (enterprise) or let us know on the forums (free users) with an example on how to interpret the output, and we can add it.

Getting latency/memory information

The profiling API expects:

• A TFLite file.

• A reference model (which model is closest to your architecture) - you can choose between gestures-large-f32, gestures-large-i8, image-32-32-mobilenet-f32, image-32-32-mobilenet-i8, image-96-96-mobilenet-f32, image-96-96-mobilenet-i8, image-320-320-mobilenet-ssd-f32, keywords-2d-f32, keywords-2d-i8. Make sure to use i8 models if you have quantized your model.

Here's how you invoke the API from Python:

Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio), but they might not be suitable for all applications. Perhaps you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing. In this tutorial you'll learn how to support these use cases by adding custom processing blocks to the studio.

There is also a complete video covering how to implement your custom DSP block:

Prerequisites

1. Building your first custom processing block

This creates a copy of the example project locally. Then, you can run the example either through Docker or locally via:

Docker

Locally

Exposing the processing block to the world

2. Install the ngrok binary for your platform.

3. Get a URL to access the processing block from the outside world via:

This yields a public URL for your block under Forwarding. Note down the address that includes https://.

Adding the custom block to Edge Impulse

Now that the custom processing block was created, and you've made it accessible to the outside world, you can add this block to Edge Impulse. In a project, go to Create Impulse, click Add a processing block, choose Add custom block (in the bottom left corner of the modal), and paste in the public URL of the block:

After you click Add block the block will show like any other processing block.

Add a learning bloc, then click Save impulse to store the impulse.

2. Adding configuration options

Processing blocks have configuration options which are rendered on the block parameter page. These could be filter configurations, scaling options, or control which visualizations are loaded. These options are defined in the parameters.json file. Let's add an option to smooth raw data. Open example-custom-processing-block-python/parameters.json and add a new section under parameters:

Then, open example-custom-processing-block-python/dsp.py and replace its contents with:

Restart the Python script, and then click Custom block in the studio (in the navigation bar). You now have a new option 'Smooth'. Every time an option changes we'll re-run the block, but as we have not written any code to respond to these changes nothing will happen.

2.1 Valid configuration types

We support a number of different types for configuration fields. These are:

• int - renders a numeric textbox that expects integers.

• float - renders a numeric textbox that expects floating point numbers.

• string - renders a textbox that expects a string.

• boolean - renders a checkbox.

• select - renders a dropdown box. This also requires the parameter valid which should be an array of valid values. E.g. this renders a dropdown box with options 'low', 'high' and 'none':

3. Implementing smoothing and drawing graphs

To show the user what is happening we can also draw visuals in the processing block. Right now we support graphs (linear and logarithmic) and arbitrary images. By showing a graph of the smoothed sample we can quickly identify what effect the smooth option has on the raw signal. Open dsp.py and replace the content with the following script. It contains a very basic smoothing algorithm and draws a graph:

Restart the script, and click the Smooth toggle to observe the difference. Congratulations! You have just created your first custom processing block.

3.1 Adding features to labels

If you extract set features from the signal, like the mean, that you that return, you can also label these features. These labels will be used in the feature explorer. To do so, add a labels array that contains strings that map back to the features you return (labels and features should have the same length).

4. Other type of graphs

In the previous step we drew a linear graph, but you can also draw logarithmic graphs or even full images. This is done through the type parameter:

4.1 Logarithmic graphs

This draws a graph with a logarithmic scale:

4.2 Images

To show an image you should return the base64 encoded image and its MIME type. Here's how you draw a small PNG image:

4.3 Dimensionality reduction

If you output high-dimensional data (like a spectrogram or an image) you can enable dimensionality reduction for the feature explorer. This will run UMAP over the data to compress the features into three dimensions. To do so, set:

On the info object in parameters.json.

4.4 Full documentation

5. Running on device

Your custom block behaves exactly the same as any of the built-in blocks. You can process all your data, train neural networks or anomaly blocks, and validate that your model works.

However, we cannot automatically generate optimized native code for the block, like we do for built-in processing blocks, but we try to help you write this code as much as possible.

In your custom DSP code, open the parameters.json file, you should have something similar to the following:

The cppType field is used to generate a function that you can implement on your custom C++ library that you get from the deployment page.

When you export your project to a C++ library we generate structures for all the configuration options in the model-parameters/dsp_blocks.h header file. You only need to implement the extract_custom_block_features function. It takes your {cppType} and generates the following extract_{cppType}_features function.

For example with the above cppType parameter:

Implement your function in the main.cpp file (or somewhere else, just make sure it is referenced).

Also, please have a look at the video on the top of this page (around minute 25) where Jan explains how to how to implement your custom DSP block in with your C++ library.

6. Other resources

7. Conclusion

With good feature extraction you can make your machine learning models smaller and more reliable, which are both very important when you want to deploy your model on embedded devices. With custom processing blocks you can now develop new feature extraction pipelines straight from Edge Impulse. Whether you're following the latest research, want to implement proprietary algorithms, or are just exploring data.

This is the specification for the deployment-metadata.json file from Building deployment blocks.

export interface DeploymentMetadataV1 {
    version: 1;
    // Global deployment counter
    deployCounter: number;
    // The output classes (for classification)
    classes: string[];
    // The number of samples to be taken per inference (e.g. 100Hz data, 3 axis, 2 seconds => 200)
    samplesPerInference: number;
    // Number of axes ((e.g. 100Hz data, 3 axis, 2 seconds => 3)
    axesCount: number;
    // Frequency of the data
    frequency: number;
    // TFLite models (already converted and quantized)
    tfliteModels: {
        // Information about the model type, e.g. quantization parameters
        details: KerasModelIODetails;
        // Name of the input tensor
        inputTensor: string;
        // Name of the output tensor
        outputTensor: string;
        // Path of the model on disk
        modelPath: string;
        // Calculated arena size when running TFLite in interpreter mode
        arenaSize: number;
        // Number of values to be passed into the model
        inputFrameSize: number;
    }[];
    // Project information
    project: {
        name: string;
        // API key, only set for deploy blocks with privileged flag and development keys set
        apiKey: string | undefined;
    };
    // Impulse information
    impulse: DeploymentMetadataImpulse;
    // Sensor guess based on the input
    sensor: 'camera' | 'microphone' | 'accelerometer' | undefined;
    // Folder locations
    folders: {
        // Input files are here, the input folder contains 'edge-impulse-sdk', 'model-parameters', 'tflite-model'
        input: string;
        // Write your output file here
        output: string;
    };
}

export type ResizeEnum = 'squash' | 'fit-short' | 'fit-long' | 'crop';
export type CropAnchorEnum = 'top-left' | 'top-center' | 'top-right' |
                             'middle-left' | 'middle-center' | 'middle-right' |
                             'bottom-left' | 'bottom-center' | 'bottom-right';

export interface CreateImpulseStateInput {
    id: number;
    type: 'time-series' | 'image';
    name: string;
    title: string;
    windowSizeMs?: number;
    windowIncreaseMs?: number;
    imageWidth?: number;
    imageHeight?: number;
    resizeMode?: ResizeEnum;
    cropAnchor?: CropAnchorEnum;
}

export interface CreateImpulseStateDsp {
    id: number;
    type: string | 'custom';
    name: string;
    axes: string[];
    title: string;
    customUrl?: string;
}

export interface CreateImpulseStateLearning {
    id: number;
    type: string;
    name: string;
    dsp: number[];
    title: string;
}

export interface CreateImpulseState {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: CreateImpulseStateDsp[];
    learnBlocks: CreateImpulseStateLearning[];
}

export interface DSPConfig {
    options: { [k: string ]: string | number | boolean };
}

export type DSPFeatureMetadataOutput = {
    type: 'image',
    shape: { width: number, height: number, channels: number }
} | {
    type: 'spectrogram',
    shape: { width: number, height: number }
} | {
    type: 'flat',
    shape: { width: number }
};

export interface DSPFeatureMetadata {
    created: Date;
    dspConfig: DSPConfig;
    labels: string[];   // the training labels
    featureLabels: string[];
    valuesPerAxis: number;
    windowCount: number;
    windowSizeMs: number;
    windowIncreaseMs: number;
    frequency: number;
    includedSamples: { id: number, windowCount: number }[];
    outputConfig: DSPFeatureMetadataOutput;
}

/**
 * Information necessary to quantize or dequantize the contents of a tensor
 */
export type KerasModelTensorDetails = {
    dataType: 'float32'
} | {
    dataType: 'int8';
    // Scale and zero point are used only for quantized tensors
    quantizationScale?: number;
    quantizationZeroPoint?: number;
};

export type KerasModelTypeEnum = 'int8' | 'float32' | 'requiresRetrain';

/**
 * Information required to process a model's input and output data
 */
export interface KerasModelIODetails {
    modelType: KerasModelTypeEnum;
    inputs: KerasModelTensorDetails[];
    outputs: KerasModelTensorDetails[];
}

export interface DeploymentMetadataImpulse {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: (CreateImpulseStateDsp & { metadata: DSPFeatureMetadata | undefined })[];
    learnBlocks: CreateImpulseStateLearning[];
}

Edge Impulse FOMO (Faster Objects, More Objects) is a novel machine learning algorithm that brings object detection to highly constrained devices. It lets you count objects, find the location of objects in an image, and track multiple objects in real-time using up to 30x less processing power and memory than MobileNet SSD or YOLOv5.

For example, FOMO lets you do 60 fps object detection on a Raspberry Pi 4:

And here's FOMO doing 30 fps object detection on an Arduino Nicla Vision (Cortex-M7 MCU), using 245K RAM.

You can find the complete Edge Impulse project with the beers vs. cans model, including all data and configuration here: https://studio.edgeimpulse.com/public/89078/latest.

How does this 🪄 work?

So how does that work? First, a small primer. Let's say you want to detect whether you see a face in front of your sensor. You can approach this in two ways. You can train a simple binary classifier, which says either "face" or "no face", or you can train a complex object detection model which tells you "I see a face at this x, y point and of this size". Object detection is thus great when you need to know the exact location of something, or if you want to count multiple things (the simple classifier cannot do that) - but it's computationally much more intensive, and you typically need much more data for it.

The design goal for FOMO was to get the best of both worlds: the computational power required for simple image classification, but with the additional information on location and object count that object detection gives us.

Heat maps

The first thing to realize is that while the output of the image classifier is "face" / "no face" (and thus no locality is preserved in the outcome) the underlying neural network architecture consists of a number of convolutional layers. A way to think about these layers is that every layer creates a diffused lower-resolution image of the previous layer. E.g. if you have a 16x16 image the width/height of the layers may be:

1. 16x16

2. 4x4

3. 1x1

Each 'pixel' in the second layer maps roughly to a 4x4 block of pixels in the input layer, and the interesting part is that locality is somewhat preserved. The 'pixel' in layer 2 at (0, 0) will roughly map back to the top left corner of the input image. The deeper you go in a normal image classification network, the less of this locality (or "receptive field") is preserved until you finally have just 1 outcome.

FOMO uses the same architecture, but cuts off the last layers of a standard image classification model and replaces this layer with a per-region class probability map (e.g. a 4x4 map in the example above). It then has a custom loss function which forces the network to fully preserve the locality in the final layer. This essentially gives you a heatmap of where the objects are.

The resolution of the heat map is determined by where you cut off the layers of the network. For the FOMO model trained above (on the beer bottles) we do this when the size of the heat map is 8x smaller than the input image (input image of 160x160 will yield a 20x20 heat map), but this is configurable. When you set this to 1:1 this actually gives you pixel-level segmentation and the ability to count a lot of small objects.

Training on centroids

A difference between FOMO and other object detection algorithms is that it does not output bounding boxes, but it's easy to go from heat map to bounding boxes. Just draw a box around a highlighted area.

However, when working with early customers we realized that bounding boxes are merely an implementation detail of other object detection networks, and are not a typical requirement. Very often the size of objects is not important as cameras are in fixed locations (and objects thus fixed size), but rather you just want the location and the count of objects.

Thus, we now train on the centroids of objects. This makes it much easier to count objects that are close (every activation in the heat map is an object), and the convolutional nature of the neural network ensures we look around the centroid for the object anyway.

A downside of the heat map is that each cell acts as its own classifier. E.g. if your classes are "lamp", "plant" and "background" each cell will be either lamp, plant, or background. It's thus not possible to detect objects with overlapping centroids. You can see this in the Raspberry Pi 4 video above at 00:18 where the beer bottles are too close together. This can be solved by using a higher resolution heat map.

Flexible and very, very fast

A really cool benefit of FOMO is that it's fully convolutional. If you set an image:heat map factor of 8 you can throw in a 96x96 image (outputs 12x12 heat map), a 320x320 image (outputs 40x40 heat map), or even a 1024x1024 image (outputs 128x128 heat map). This makes FOMO incredibly flexible, and useful even if you have very large images that need to be analyzed (e.g. in fault detection where the faults might be very, very small). You can even train on smaller patches, and then scale up during inference.

Additionally FOMO is compatible with any MobileNetV2 model. Depending on where the model needs to run you can pick a model with a higher or lower alpha, and transfer learning also works (although you need to train your base models specifically with FOMO in mind). This makes it easy for end customers to use their existing models and fine-tune them with FOMO to also add locality (e.g. we have customers with large transfer learning models for wildlife detection).

Together this gives FOMO the capabilities to scale from the smallest microcontrollers all the way to full gateways or GPUs. Just some numbers:

1. The video on the top classifies 60 times / second on a stock Raspberry Pi 4 (160x160 grayscale input, MobileNetV2 0.1 alpha). This is 20x faster than MobileNet SSD which does ~3 frames/second.

2. The second video on the top classifies 30 times / second on an Arduino Nicla Vision board with a Cortex-M7 MCU running at 480MHz) in ~240K of RAM (96x96 grayscale input, MobileNetV2 0.35 alpha).

3. During Edge Impulse Imagine we demonstrated a FOMO model running on a Himax WE-I Plus doing 14 frames per second on a DSP (video). This model ran in under 150KB of RAM (96x96 grayscale input, MobileNetV2 0.1 alpha). [1]

4. The smallest version of FOMO (96x96 grayscale input, MobileNetV2 0.05 alpha) runs in <100KB RAM and ~10 fps. on a Cortex-M4F at 80MHz. [1]

[1] Models compiled using EON Compiler.

How to get started?

To build your first FOMO models:

1. Create a new project in Edge Impulse.

2. Make sure to set your labeling method to 'Bounding boxes (object detection)'.

3. Collect and prepare your dataset as in Object detection

4. Add an 'Object Detection (Images)' block to your impulse.

5. Under Images, select 'Grayscale'

6. Under Object detection, select 'Choose a different model' and select one of the FOMO models.

7. Make sure to lower the learning rate to 0.001 to start.

FOMO is currently compatible with all fully-supported development boards that have a camera, and with Edge Impulse for Linux (any client). Of course, you can export your model as a C++ Library and integrate it as usual on any device or development board, the output format of models is compatible with normal object detection models; and our SDK runs on almost anything under the sun (see Running your impulse locally for an overview) from RTOS's to bare-metal to special accelerators and GPUs.

Expert mode tips

Additional configuration for FOMO can be accessed via expert mode.

Object weighting

FOMO is sensitive to the ratio of objects to background cells in the labelled data. By default the configuration is to weight object output cells x100 in the loss function, object_weight=100, as a way of balancing what is usually a majority of background. This value was chosen as a sweet spot for a number of example use cases. In scenarios where the objects to detect are relatively rare this value can be increased, e.g. to 1000, to have the model focus even more on object detection (at the expense of potentially more false detections).

MobileNet cut point

FOMO uses MobileNetV2 as a base model for its trunk and by default does a spatial reduction of 1/8th from input to output (e.g. a 96x96 input results in a 12x12 output). This is implemented by cutting MobileNet off at the intermediate layer block_6_expand_relu

Choosing a different cut_point results in a different spatial reduction; e.g. if we cut higher at block_3_expand_relu FOMO will instead only do a spatial reduction of 1/4 (i.e. a 96x96 input results in a 24x24output)

Note though; this means taking much less of the MobileNet backbone and results in a model with only 1/2 the params. Switching to a higher alpha may counteract this parameter reduction. Later FOMO releases will counter this parameter reduction with a UNet style architecture.

FOMO classifier capacity

FOMO can be thought of logically as the first section of MobileNetV2 followed by a standard classifier where the classifier is applied in a fully convolutional fashion.

In the default configuration this FOMO classifier is equivalent to a single dense layer with 32 nodes followed by a classifier with num_classes outputs.

For a three way classifier, using the default cut point, the result is a classifier head with ~3200 parameters.

 LAYER                          SHAPE                NUMBER OF PARAMETERS
 block_6_expand_relu (ReLU)     (None, 20, 20, 96)   0                                         
 head (Conv2D)                  (None, 20, 20, 32)   3104                                            
 logits (Conv2D)                (None, 20, 20, 3)    99

We have the option of increasing the capacity of this classifier head by either 1) increasing the number of filters in the Conv2D layer, 2) adding additional layers or 3) doing both.

For example we might change the number of filters from 32 to 16, as well as adding another convolutional layer, as follows.

 LAYER                          SHAPE                NUMBER OF PARAMETERS
 block_6_expand_relu (ReLU)     (None, 20, 20, 96)   0
 head_1 (Conv2D)                (None, 20, 20, 16)   1552                                         
 head_2 (Conv2D)                (None, 20, 20, 16)   272                                          
 logits (Conv2D)                (None, 20, 20, 3)    51

For some problems an additional layer can improve performance, and in this case actually uses less parameters. It can though potentially take longer to train and require more data. In future releases the tuning of this aspect of FOMO can be handled by the EON Tuner.

Performance and Minimum Requirements

Just like the rest of our Neural Network-based learning blocks, FOMO is delivered as a set of basic math routines free of runtime dependencies. This means that there are virtually no limitations to running FOMO, other than:

1. Making sure the model itself can fit into the target's memory (flash/RAM), and

2. making sure the target also has enough memory to hold the image buffer (flash/RAM)in addition to your application logic

In all, we have seen buffer, model and app logic (including wireless stack) fit in as little as 200KB for 64x64 pixel images. But we would definitely recommend a target with at least 512KB so that you can take advantage of larger image sizes and a wider range of model optimizations.

With regards to latency, the speed of the target will determine the maximum number of frames that can be processed in a given interval (fps). This will of course be influenced by any other tasks the CPU may need to complete, but we have consistently seen MCUs running @ 80MHz complete a full pass on a 64x64 pixel image in under one second, which should translate to just under 1fps once you add the rest of your app logic. Keep in mind that frame throughput can increase dramatically at higher speeds or when tensor acceleration is available. We have measured 40-60 fps consistently on a Raspberry Pi 4 and ~15 fps on unaccelerated 480MHz targets. The table below summarizes this trade-off:

After training and validating your model, you can now deploy it to any device. This makes the model run without an internet connection, minimizes latency, and runs with minimal power consumption.

The Deployment page consists of a variety of deploy options to choose from depending on your target device. Regardless of whether you are using a fully supported development board or not, Edge Impulse provides deploy options through C++ library in which you can use to deploy your model on any targets (as long as the target has enough compute can handle the task).

The following are the 5 main categories of deploy options currently supported by Edge Impulse:

1. Deploy as a customizable library

2. Deploy as a pre-built firmware - for fully supported development boards

3. Run directly on your phone or computer

4. Use Edge Impulse for Linux for Linux targets

5. Create a custom deployment block (Enterprise feature)

Deploying as a customizable library

This deploy option lets you turn your impulse into a fully optimized source code that can be further customized and integrated with your application. This option supports the following libraries:

Arduino Library

You can run your impulse locally as an Arduino library. This packages all of your signal processing blocks, configuration and learning blocks up into a single package.

To deploy as an Arduino library, select Arduino library on the Deployment page and click Build to create the library. Download the .ZIP file and import it as a sketch in your Arduino IDE then run your application.

For a full tutorial on how to run your impulse locally as an arduino library, have a look at Running your impulse locally - Arduino.

C++ Library

You can run your Impulse as a C++ library. This packages all of your signal processing blocks, configuration and learning blocks up into a single package that can be easily ported to your custom applications.

Visit Running your impulse locally for a deep dive on how to deploy your impulse as a C++ library.

Cube.MX CMSIS-PACK library

If you want to deploy your impulse to an STM32 MCU, you can use the Cube.MX CMSIS-PACK. This packages all your signal processing blocks, configuration and learning blocks up into a single package. You can include this package in any STM32 project with a single function call.

Have a look at Running your impulse locally - using CubeAI for a deep dive on how to deploy your impulse on STM32 based targets using the Cube.MX CMSIS-PACK.

WebAssembly Library

When you want to deploy your impulse to a web app you can use the WebAssembly library.This packages all your signal processing blocks, configuration and learning blocks up into a single package that can run without any compilation.

Have a look at Running your impulse locally - through WebAssembly (Browser) fora deep dive on how you can run your impulse to classify sensor data in your Node.js application.

Deploy as a pre-built firmware

For this option, you can use a ready-to-go binary for your development board that bundles signal processing blocks, configuration and learning blocks up into a single package. This option is currently only available for fully supported development boards as shown in the image below:

To deploy your model using ready to go binaries, select your target device and click "build". Flash the downloaded firmware to your device then run the following command:

edge-impulse-run-impulse

The impulse runner shows the results of your impulse running on your development board. This only applies to ready-to-go binaries built from the studio.

Edge Impulse for Linux

If you are developing for Linux based devices, you can use Edge Impulse for Linux for deployment. It contains tools which let you collect data from any microphone or camera, can be used with the Node.js, Python, Go and C++ SDKs to collect new data from any sensor, and can run impulses with full hardware acceleration - with easy integration points to write your own applications.

For a deep dive on how to deploy your impulse to linux targets using Edge Impulse for linux, you can visit the Edge Impulse for Linux tutorial.

Deploy to your mobile phone/computer

You can run your impulse directly on your computer/mobile phone without the need of additional app. To run on your computer, you simply just need to select "computer" then click "Switch to classification mode". To run on your mobile phone, select 'Mobile Phone' then scan the QR code and click 'switch to classification mode".

Optimizations

Enabling EON Compiler

When building your impulse for deployment, Edge Impulse gives you the option of adding another layer of optimization to your impulse using the EON compiler. The EON Compiler lets you run neural networks in 25-55% less RAM, and up to 35% less flash, while retaining the same accuracy, compared to TensorFlow Lite for Microcontrollers.

To activate the EON Compiler, select you preferred deployment option then go to Enable EON™ Compiler then enable it and click 'Build' to build your impulse for deployment.

To have a peek of how your impulse would utilize compute resources of your target device, Edge Impulse also gives an estimate of latency, flash, RAM to be consumed by your target device even before deploying your impulse locally. This can really save you a lot of engineering time costs incurred by recurring iterations and experiments.

You can also select whether to run the unquantized float32 or the quantized int8 models as shown in the image below.

The above confusion matrix is only based on the test data to help you know how your model performs on unseen real world data. It can also help you know whether your model has learned to overfit on your training data which is a common occurrence.

Training and deploying high performing ML models is usually considered as a continuous process rather than a one time exercise. When you are validating your model and discover an overfit, you might consider adding some more diverse data then perform model retraining while maintaining the initially set DSP and Neural Network block configurations.

Also during inference If you find that the data distribution has drifted significantly from the initial training distribution, it is usually a good common practice to retrain your model on the newer data distribution to keep up with the high model performance.

The Retrain model feature in the Edge Impulse Studio is useful when adding new data to your project. It uses already known parameters from your selected DSP and ML blocks then uses them to automatically regenerate new features and retrain the Neural Network model in one single step. You can consider this a shortcut for retraining your model since you don’t need to go through all the blocks in your impulse one by one again.

To retrain your model after adding some data, navigate to the Retrain model tab and click Train model.

Building data pipelines is a very useful feature where you can stack several transformation blocks similar to the . They can be used in a standalone mode (just execute several transformation jobs in a pipeline), to feed a dataset or to feed a project.

The examples in the screenshots below shows how to create and use a pipeline to create the 'AMS Activity 2022' dataset.

Create a pipeline

To create a new pipeline, click on '+Add a new pipeline:

Get the steps from your transformation blocks

In your organization workspace, go to Custom blocks -> Transformation and select Run job on the job you want to add.

Select Copy as pipeline step and paste it to the configuration json file.

You can then paste the copied step directly to the respected field.

Below, you have an option to feed the data to either a organisation dataset or an Edge Impulse project

Schedule and notify

By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.

Once the pipeline has successfully finished, it can send an email to the Users to notify.

Run the pipeline

Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.

Run the pipeline from the UI

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.

Run the pipeline from code

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.

Webhooks

Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:

Overview

* Maybe not available on the board and require to attach additional sensors, see the dedicated page for more information.

** RAM used for the latency calculation, may differ from the datasheet.

*** ROM used for the latency calculation, may differ from the datasheet.

The series of fusion processors from utilize ARM's low power Cortex-M55 CPUs with dedicated Ethos-U55 microNPUs to run embedded ML workloads quickly and efficiently. The devices feature both 'High Power' cores designed for large model architectures, as well as 'High Efficiency' cores designed for low power continuous monitoring. The and are both fully supported by Edge Impulse. The Ensemble E7 kits feature multiple core types, dual MEMS microphones, accelerometers, and a MIPI camera interface.

To get started with the Alif Ensemble E7 and Edge Impulse you'll need the following hardware:

• OR

Installing dependencies

To set this device up in Edge Impulse, you will need to install the following software:

2. The latest Alif Security Toolkit:

• Extract the archive, and read through the included Security Toolkit Quick Start Guide to finalize the installation

Connecting to Edge Impulse

Once you have installed it's time to connect the development board to Edge Impulse.

1. Configure your hardware

To interface the Alif Ensemble E7 AppKit or Development Kit, you'll need to make sure your hardware is properly configured and connected to your computer. Follow the steps below to prepare your specific kit for connection to Edge Impulse

2. Flash the default firmware to the device

After configuring the hardware, the next step is to flash the default Edge Impulse Firmware. This will allow us to collect data directly from your Ensemble device. To update the firmware:

2. Navigate to the directory where you installed the Alif Security Toolkit

3. Copy the .bin files from the Edge Impulse firmware directory into the build/images directory of the Alif Security Toolkit

4. Copy all .json files from the Edge Impulse firmware directory into the build/config directory of the Alif Security Toolkit

5. From a command prompt or terminal, run the following commands:

3. Run the Edge Impulse CLI!

Now, the Ensemble device can connect to the Edge Impulse CLI installed earlier. To test the CLI for the first time, either:

OR

Then, from a command prompt or terminal on your computer, run:

This will start a wizard which will ask you to log in and choose an Edge Impulse project. You should see your new or cloned project listed on the command line. Use the arrow keys and hit Enter to select your project.

5. Verifying that the device is connected

That's all! Your device is now connected to Edge Impulse. To verify this, go to

Next steps: building a machine learning model

With everything set up you can now build your first machine learning model with these tutorials. This will walk you through the process of collecting data and training a new ML model:

Alternatively, you can test on-device inference with a demo model included in the base firmware binary. To do this, you may run the following command from your terminal:

Then, once you've tested out training and deployment with the Edge Impulse Firmware, learn how to integrate impulses with your own custom Ensemble based application:

This is a list of development boards that are fully supported by Edge Impulse. These boards come with a special firmware which enables data collection from all their sensors, allows you to build new ready-to-go binaries that include your trained impulse, and come with examples on integrating your impulse with your custom firmware. These boards are the perfect way to start building machine learning solutions on real embedded hardware.

Different development board or custom PCB? No problem! You can upload data to Edge Impulse in a variety of ways, such as using the , the SDK, or by (e.g. CSV, JPG, WAV).

From there, your trained model can be deployed as a . It requires some effort, but most build systems (for computers, smartphones, and microcontrollers) will work with our C++ library. This, of course, requires that your build system has a C++ compiler and that there is enough flash/RAM on your device to run the library/model. Also, if you feel like porting the official Edge Impulse firmware to your own board, use this .

Just want to experience Edge Impulse? You can also use your !

Officially supported MCU targets

Officially supported CPU/GPU targets

Officially supported AI accelerators

Production targets

Community targets