Welcome to Edge Impulse! We enable developers to create the next generation of intelligent device solutions with embedded Machine Learning. In the documentation you'll find user guides, tutorials and API documentation. For support, visit the forums.

Get started with any device

Follow these three steps to build your first embedded Machine Learning model - no worries, you can use almost any device to get started.

1. You'll need some data:

   • If you have an existing development board or device, you can collect data with a few lines of code using the Data forwarder or the Edge Impulse for Linux SDK.

   • If you want to collect live data from a supported development kit, select your board from the list of fully supported development boards and follow the instructions to connect your board to edge impulse.

   • If you already have a dataset, you can upload it via the Uploader.

   • If you have a mobile phone you can use it as a sensor to collect data, see Mobile phone.

2. Try the tutorials on continuous motion recognition, responding to your voice, recognizing sounds from audio, adding sight to your sensors or object detection. These will let you build machine learning models that detect things in your home or office.

3. After training your model you can run your model on your device:

   • If you want to integrate the model with your own firmware or project you can export your complete model (including all signal processing code and machine learning models) to a C++ or Arduino library with no external dependencies (open source and royalty-free), see Running your impulse locally.

   • If you have a fully supported development board (or your mobile phone) you can build new firmware - which includes your model - directly from the UI. It doesn't get easier than that!

   • If you have a gateway, a computer or a web browser where you want to run your model, you can export to WebAssembly and run it anywhere you can run JavaScript.

Suitable for any type of embedded ML application

We have some great tutorials, but you have full freedom in the models that you design in Edge Impulse. You can plug in new signal processing blocks, and completely new neural networks. See Building custom processing blocks, or click the three dots on a neural network page and select 'Switch to Keras (expert) mode'.

API Documentation

You can access any feature in the Edge Impulse Studio through the Edge Impulse API. We also have the Ingestion service if you want to send data directly, and we have an open Remote management protocol to control devices from the Studio.

Enterprise version

For larger teams, and companies with lots of data we offer an enterprise version of Edge Impulse. The enterprise version offers team collaboration on projects, a dataset builder that makes your internal data available to your whole team, integrations with your cloud buckets, transformation blocks that let you extract ML features from thousands of files in one go, and custom processing and deployment blocks for your organization. You can find documentation under Organizations or contact us via hello@edgeimpulse.com for more information.

Machine learning (ML) is a way of writing computer programs. Specifically, it’s a way of writing programs that process raw data and turn it into information that is meaningful at an application level.

For example, one ML program might be designed to determine when an industrial machine has broken down based on readings from its various sensors, so that it can alert the operator. Another ML program might take raw audio data from a microphone and determine if a word has been spoken, so it can activate a smart home device.

Unlike normal computer programs, the rules of ML programs are not determined by a developer. Instead, ML uses specialized algorithms to learn rules from data, in a process known as training.

In a traditional piece of software, an engineer designs an algorithm that takes an input, applies various rules, and returns an output. The algorithm’s internal operations are planned out by the engineer and implemented explicitly through lines of code. To predict breakdowns in an industrial machine, the engineer would need to understand which measurements in the data indicate a problem and write code that deliberately checks for them.

This approach works fine for many problems. For example, we know that water boils at 100°C at sea level, so it’s easy to write a program that can predict whether water is boiling based on its current temperature and altitude. But in many cases, it can be difficult to know the exact combination of factors that predicts a given state. To continue with our industrial machine example, there might be various different combinations of production rate, temperature, and vibration level that might indicate a problem but are not immediately obvious from looking at the data.

To create an ML program, an engineer first collects a substantial set of training data. They then feed this data into a special kind of algorithm, and let the algorithm discover the rules. This means that as ML engineers, we can create programs that make predictions based on complex data without having to understand all of the complexity ourselves.

Through the training process, the ML algorithm builds a model of the system based on the data we provide. We run data through this model to make predictions, in a process called inference.

Where can machine learning help?

Machine learning is an excellent tool for solving problems that involve pattern recognition, especially patterns that are complex and might be difficult for a human observer to identify. ML algorithms excel at turning messy, high-bandwidth raw data into usable signals, especially combined with conventional signal processing.

For example, the average person might struggle to recognize the signs of a machine failure given ten different streams of dense, noisy sensor data. However, a machine learning algorithm can often learn to spot the difference.

But ML is not always the best tool for the job. If the rules of a system are well defined and can be easily expressed with hard-coded logic, it’s usually more efficient to work that way.

What is embedded ML?

Recent advances in microprocessor architecture and algorithm design have made it possible to run sophisticated machine learning workloads on even the smallest of microcontrollers. Embedded machine learning, also known as TinyML, is the field of machine learning when applied to embedded systems such as these.

Bandwidth—ML algorithms on edge devices can extract meaningful information from data that would otherwise be inaccessible due to bandwidth constraints.

Latency—On-device ML models can respond in real-time to inputs, enabling applications such as autonomous vehicles, which would not be viable if dependent on network latency.

Economics—By processing data on-device, embedded ML systems avoid the costs of transmitting data over a network and processing it in the cloud.

Reliability—Systems controlled by on-device models are inherently more reliable than those which depend on a connection to the cloud.

Privacy—When data is processed on an embedded system and is never transmitted to the cloud, user privacy is protected and there is less chance of abuse.

Learn more

After creating your Edge Impulse Studio project, you will be directed to the project's dashboard. The dashboard gives a quick overview of your project such as your project ID, the number of devices connected, the amount of data collected, the preferred labeling method, among other editable properties. You can also enable some additional capabilities to your project such as collaboration, making your project public, and showcasing your public projects using Markdown READMEs as we will see.

The figure below shows the various sections and widgets of the dashboard that we will cover here.

1. Showcasing your public projects with Markdown READMEs

The project README enables you to explain the details of your project in a short way. Using this feature, you can add visualizations such as images, GIFs, code snippets, and text to your project in order to bring your colleagues and project viewers up to speed with the important details of your project. In your README you might want to add things like:

• What the project does

• Why the project is useful

• Motivations of the project

• How to get started with the project

• What sensors and target deployment devices you used

• How you plan to improve your project

• Where users can get help with your project

To create your first README, navigate to the "about this project" widget and click "add README"

For more README inspiration, check out the public Edge Impulse project tutorials below:

2. Making your project public

To share your private project with the world, and click Make this project public.

By doing this, all of your data, block configurations, intermediate results, and final models will be shared with the world. Your project will be publicly accessible and can be cloned with a single click with the provided URL:

3. Collaboration

To add a collaborator, go to your project's dashboard and find the "Collaborators" widget. Click the '+' icon and type the username or e-mail address of the other user. The user will be invited to create an Edge Impulse account if it doesn't exist.

The user will be automatically added to the project and will get an email notification inviting them to start contributing to your project. To remove a user, simply click on the three dots besides the user then tap ‘Delete’ and they will be automatically removed.

4. Project info

The project info widget shows the project's specifications such as the project ID, labeling method, and latency calculations for your target device.

• On the labeling method dropdown, you need to specify the type of labeling your dataset and model expect. This can be either one label per data item or bounding boxes. Bounding boxes only work for object detection tasks in the studio. Note that if you interchange the labeling methods, learning blocks will appear to be hidden when building your impulse.

• One of the amazing Edge Impulse superpowers is the latency calculation component. This is an approximate time in milliseconds that the trained model and DSP operations are going to take during inference based on the selected target device. This hardware in the loop approach ensures that the target deployment device compute resources are not underutilized or over-utilized. It also saves developers' time associated with numerous inference iterations back and forth the studio in search of optimum models.

5. Block Outputs

In the Block Output section, you can download the results of the DSP and ML operations of your impulse.

The downloadable assets include the extracted features, Tensorflow SavedModel, and both quantized and unquantized TensorFlow lite models. This is particularly helpful when you want to perform other operations to the output blocks outside the Edge Impulse studio. For example, if you need a TensorflowJS model, you will just need to download the TensorFlow saved model from the dashboard and convert it to TensorFlowJS model format to be served on a browser.

6. Performance Settings

This section consists of editable parameters that directly affect the performance of the studio when building your impulse. Depending on the selected or available settings, your jobs can either be fast or slow.

The use of GPU for training and Parallel DSP jobs is currently an internal experimental feature that will be soon released.

7. Administrative Zone

To bring even more flexibility in projects, the administrative zone gives developers the power to enable other additional features that are not found in edge impulse projects by default. Most of these features are usually advanced features intended for organizations or sometimes experimental features.

To activate these features you just need to check the boxes against the specific features you want to use and click save experiments.

8. Danger Zone

The danger zone widget consists of irrevocable actions that let you to:

• Delete your project. This action removes all devices, data, and impulses from your project.

• Delete all data in this project.

• Perform train/test split. This action re-balances your dataset by splitting all your data automatically between the training and testing set and resets the categories for all data

• Launch the getting started wizard. This will remove all data, and clear out your impulse.

All collected data for each project can be viewed on the Data acquisition tab. You can see how your data has been split for train/test set as well as the data distribution for each class in your dataset. You can also send new sensor data to your project either by file upload, WebUSB, Edge Impulse API, or Edge Impulse CLI.

Add data to your project

Record new data

The panel on the right allows you to collect data directly from any fully supported platform:

• Through WebUSB.

• Using the Edge Impulse CLI daemon.

• From the Edge Impulse for Linux CLI.

The WebUSB and the Edge Impulse daemon work with any fully supported device by flashing the pre-built Edge Impulse firmware to your board. See the list of fully supported boards.

When using the Edge Impulse for Linux CLI, run edge-impulse-linux --clean and it will add your platform to the device list of your project. You will then will be able to interact with it from the Record new data panel.

Other methods

• Studio uploader.

• CLI uploader.

• CLI data forwarder.

• Ingestion API.

• Import from S3 buckets (Enterprise feature).

• Upload portals (Enterprise feature).

Dataset train/test split ratio

The train/test split is a technique for training and evaluating the performance of a machine learning algorithms. It indicates how your data is split between training and testing samples. For example, an 80/20 split indicates that 80% of the dataset is used for model training purposes while 20% is used for model testing.

This section also shows how your data samples in each class are distributed to prevent imbalanced datasets which might introduce bias during model training.

Data acquisition filter

Manually navigating to some categories of data can be time consuming, especially when dealing with a large dataset. The data acquisition filter enables the user to filter data samples based on some criteria of choice. This can be based on:

• Label - class to which a sample represents.

• Sample name - unique ID representing a sample.

• Signature validity

• Enabled and disabled samples

• Length of sample - duration of a sample.

The filtered samples can then be manipulated by editing label, deleting, moving from trains set to test set and vise versa a shown in the image above.

The data manipulations above can also be applied at the data sample level just by simply navigating to the individual data sample then clicking ⋮ and selecting the type of action you might want to perform to the specific sample. This might be renaming , editing its label, disabling, cropping, splitting, downloading, and even deleting the sample when desired.

Cropping samples

To crop a data sample, go to the sample you want to crop and click ⋮, then select Crop sample. You can specific a length, or drag the handles to resize the window, then move the window around to make your selection.

Made a wrong crop? No problem, just click Crop sample again and you can move your selection around. To undo the crop, just set the sample length to a high number, and the whole sample will be selected again.

Splitting data sample

Besides cropping you can also split data automatically. Here you can perform one motion repeatedly, or say a keyword over and over again, and the events are detected and can be stored as individual samples. This makes it easy to very quickly build a high-quality dataset of discrete events. To do so head to Data acquisition, record some new data, click, and select Split sample. You can set the window length, and all events are automatically detected. If you're splitting audio data you can also listen to events by clicking on the window, the audio player is automatically populated with that specific split.

Samples are automatically centered in the window, which might lead to problems on some models (the neural network could learn a shortcut where data in the middle of the window is always associated with a certain label), so you can select "Shift samples" to automatically move the data a little bit around.

Splitting data is - like cropping data - non-destructive. If you're not happy with a split just click Crop sample and you can move the selection around easily.

Labelling Queue

The labelling queue will only appear on your data acquisition page if you are dealing with an object detection tasks. The labelling queue shows a list of images that have been staged for annotation for your project.

If you are not dealing with an object detection task, you can simply disable the labelling queue bar by going to Dashboard > Project info > Labeling method and clicking the dropdown and selecting "one label per data item" as shown in the image below.

For more information about the labelling queue and how to perform data annotation using AI assisted labelling on Edge Impulse, you can have a look at our documentation here.

How can I share my Edge Impulse project?

The enterprise version of Edge Impulse offers on projects, go to Dashboard, find the Collaborators section, and click the '+' icon. If you have an interesting research or community project we can enable collaboration on the free version of Edge Impulse as well, by emailing hello@edgeimpulse.com.

You can also create a public version of your Edge Impulse project. This makes your project available to the whole world - including your data, your impulse design, your models, and all intermediate information - and can easily be cloned by anyone in the community. To do so, go to Dashboard, and click Make this project public.

What are the minimum hardware requirements to run the Edge Impulse inferencing library on my embedded device?

What frameworks does Edge Impulse use to train the machine learning models?

We use a wide variety of tools, depending on the machine learning model. For neural networks we typically use TensorFlow and Keras, for object detection models we use TensorFlow with Google's Object Detection API, and for 'classic' non-neural network machine learning algorithms we mainly use sklearn. For neural networks you can see (and modify) the Keras code by clicking ⋮, and selecting Switch to expert mode.

Is there a downside to enabling the EON Compiler?

By disabling EON we place the full neural network (architecture and weights) into ROM, and load it on demand. This increases memory usage, but you could just update this section of the ROM (or place the neural network in external flash, or on an SD card) to make it easier to update.

Can I use a model that has been trained elsewhere in Edge Impulse?

You cannot import a pretrained model, but you can import your model architecture and then retrain. Add a neural network block to your impulse, go to the block, click ⋮, and select Switch to expert mode. You then have access to the full Keras API.

How does the feature explorer visualize data that has more that 3 dimensions?

Does Edge impulse integrate with other cloud services?

Yes. The enterprise version of Edge Impulse can integrate directly with your cloud service to access and transform data.

What is the typical power consumption of the Edge Impulse machine learning processes on my device?

Simple answer: To get an indication of time per inference we show performance metrics in every DSP and ML block in the Studio. Multiply this by active power consumption of your MCU to get an indication of power cost per inference.

More complicated answer: It depends. Normal techniques to conserve power still apply to ML, so try to do as little as possible (do you need to classify every second, or can you do it once a minute?), be smart about when to run inference (can there be an external trigger like a motion sensor before you run inference on a camera?), and collect data in a lower power mode (don't run at full speed when sampling low resolution data, and see if your sensor can use an interrupt to wake your MCU - rather than polling).

What is the .eim model format for Edge Impulse for Linux?

How is the labeling of the data performed?

If you are working on an object detection project, you will most likely see "labelling queue" bar on your data acquisition page. The labeling queue shows you all the data that has not been labelled in your dataset.

Can't see the labeling queue? Go to Dashboard, and under 'Project info > Labeling method' select 'Bounding boxes (object detection)'.

In object detection, labelling is the process of adding a bounding box around specific objects in an image so that your machine learning model can learn and infer from it. Edge impulse studio has an inbuilt data annotation tool with AI assisted labelling to assist you in your labelling workflows as we will see.

In the Edge Impulse studio, labelling your data is as easy as dragging a box around the object, then entering a label and saving as shown below.

However, as simple the manual labelling process might look like, it sometimes can become tedious and time consuming especially when dealing with huge datasets. To make your life easier, Edge Impulse studio has an inbuilt AI-assisted labelling feature to automatically assist you in your labelling workflows.

AI Assisted labelling

there are 3 ways you can use to perform AI assisted labelling on the Edge Impulse Studio:

• Using yolov5

• Using your own model

• Using object tracking

Using YoloV5

By utilizing an existing library of pre-trained object detection models from YOLOv5 (trained with the COCO dataset), common objects in your images can quickly be identified and labeled in seconds without needing to write any code!

To label your objects with YOLOv5 classification, click the Label suggestions dropdown and select “Classify using YOLOv5.” If your object is more specific than what is auto-labeled by YOLOv5, e.g. “coffee” instead of the generic “cup” class, you can modify the auto-labels to the left of your image. These modifications will automatically apply to future images in your labeling queue.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes!

Using your own model

You can also use your own trained model to predict and label your new images. From an existing (trained) Edge Impulse object detection project, upload new unlabeled images from the Data Acquisition tab. Then, from the “Labeling queue”, click the Label suggestions dropdown and select “Classify using ”:

You can also upload a few samples to a new object detection project, train a model, then upload more samples to the Data Acquisition tab and use the AI-Assisted Labeling feature for the rest of your dataset. Classifying using your own trained model is especially useful for objects that are not in YOLOv5, such as industrial objects, etc.

Click Save labels to move on to your next raw image, and see your fully labeled dataset ready for training in minutes using your own pre-trained model!

Using Object tracking

If you have objects that are a similar size or common between images, you can also track your objects between frames within the Edge Impulse Labeling Queue, reducing the amount of time needed to re-label and re-draw bounding boxes over your entire dataset.

Draw your bounding boxes and label your images, then, after clicking Save labels, the objects will be tracked from frame to frame:

Now that your object detection project contains a fully labeled dataset, learn how to train and deploy your model to your edge device: check out our tutorial!

We are excited to see what you build with the AI-Assisted Labeling feature in Edge Impulse, please post your project on our forum or tag us on social media, @Edge Impulse!

The data explorer is a visual tool to explore your dataset, find outliers or mislabeled data, and to help label unlabeled data. The data explorer first tries to extract meaningful features from your data (through signal processing and neural network embeddings) and then uses a dimensionality reduction algorithm to map these features to a 2D space. This gives you a one-look overview of your complete dataset.

Using the data explorer

To access the data explorer head to Data acquisition, click Data explorer, then select a way to generate the data explorer. Depending on you data you'll see three options:

• Using a pre-trained model - here we use a large neural network trained on a varied dataset to generate the embeddings. This works very well if you don't have any labeled data yet, or want to look at new clusters of data. This option is available for keywords and for images.

• Using your trained impulse - here we use the neural network block in your impulse to generate the embeddings. This typically creates even better visualizations, but will fail if you have completely new clusters of data as the neural network hasn't learned anything about them. This option is only available if you have a trained impulse.

• Using the preprocessing blocks in your impulse - here we skip the embeddings, and just use your selected signal processing blocks to create the data explorer. This creates a similar visualization as the feature explorer but in a 2D space and with extra labeling tools. This is very useful if you don't have any labeled data yet, or if you have new clusters of data that your neural network hasn't learned yet.

Then click Generate data explorer to create the data explorer. If you want to make a different choice after creating the data explorer click ⋮ in the top right corner and select Clear data explorer.

Viewing and modifying data

To view an item in your dataset just click on any of the dots (some basic information appears on hover). Information about the sample, and a preview of the data item appears at the bottom of the data explorer. You can click Set label (or l on your keyboard) to set a new label for the data item, or press Delete item (or d on your keyboard) to remove the data item. These changes are queued until you click Save labels (at the top of the data explorer).

Assisted labeling

The data explorer marks unlabeled data in gray (with an 'Unlabeled' label). To label this data you click on any gray dot. To then set a label by clicking the Set label button (or by pressing l on your keyboard) and enter a label. Other unlabeled data in the vicinity of this item will automatically be labeled as well. This way you can quickly label clustered data.

To upload unlabeled data you can either:

• Use the upload UI and select the 'Leave data unlabeled' option.

• Select the items in your dataset under Data acquisition, select all relevant items, click Edit labels and set the label to an empty string.

• When uploading data through the ingestion API, set the x-no-label header to 1, and the x-label to an empty string.

Or, if you want to start from scratch, click the three dots on top of the data explorer, and select Clear all labels.

Wait, how does this work?

The data explorer uses a three-stage process:

1. It runs your data through an input and a DSP block - like any impulse.

2. It passes the result of 1) through part of a neural network. This forces the neural network to compress the DSP output even further, but to features that are highly specialized to distinguish the exact type of data in your dataset (called 'embeddings').

3. The embeddings are passed through t-SNE, a dimensionality reduction algorithm.

So what are these embeddings actually? Let's imagine you have the model from the Continuous motion recognition tutorial. Here we slice data up in 2-second windows and run a signal processing step to extract features. Then we use a neural network to classify between motions. This network consists of:

• 33 input features (from the signal processing step)

• A layer with 20 neurons

• A layer with 10 neurons

• A layer with 4 neurons (the number of different classes)

While training the neural network we try to find the mathematical formula that best maps the input to the output. We do this by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step - just like our signal processing step - but highly tuned for your specific data. For example, in the first layer, it'll learn what features are correlated, in the second it derives new features, and in the final layer, it learns how to distinguish between classes of motions.

In the data explorer we now cut off the final layer of the neural network, and thus we get the derived features back - these are called "embeddings". Contrary to features we extract using signal processing we don't really know what these features are - they're specific to your data. In essence, they provide a peek into the brain of the neural network. Thus, if you see data in the data explorer that you can't easily separate, the neural network probably can't either - and that's a great way to spot outliers - or if there's unlabeled data close to a labeled cluster they're probably very similar - great for labeling unknown data!

Examples of different embeddings

Here's an example of using the data explorer to visualize a very complex computer vision dataset (distinguishing between the four cats of one of our infrastructure engineers).

No embeddings (just running t-SNE over the images)

With embeddings from a pretrained MobileNetV2 model

With embeddings from a custom ML model

For less complex datasets, or lower-dimensional data you'll typically see more separation, even without custom models.

Questions? Excited?

If you have any questions about the data explorer or embeddings, we'd be happy to help on the forums or reach out to your solutions engineer. Excited? Talk to us to get access to the data explorer, and finally be able to label all that sensor data you've collected!

After collecting data for your project, you can now create your Impulse. A complete Impulse will consist of 3 main building blocks: input block, processing block and a learning block.

This view is one of the most important, here you will build your own machine learning pipeline.

Impulse example for movement classification using accelerometer data:

Impulse example for object detection using images:

Input block

The input block indicates the type of input data you are training your model with. This can be time series (audio, vibration, movements) or images.

Time series (audio, vibration, movements)

• The input axes field lists all the axis referenced from your training dataset

• The window size is the size of the raw features that is used for the training

• The window increase is used to artificially create more features (and feed the learning block with more information)

• The frequency is automatically calculated based on your training samples. You can modify this value but you currently cannot use values lower than 0.000016 (less than 1 sample every 60s).

• Zero-pad data: Adds zero values when raw feature is missing

Below is a sketch to summarize the role of each parameters:

Images

• Axes: Images

• Image width & height: Most of our pre-trained models work with square images.

• Resize mode: You have three options, Squash, Fit to the shortest axis, Fit to the longest axis

Processing Block

A processing block is basically a feature extractor. It consists of DSP (Digital Signal Processing) operations that are used to extract features that our model learns on. These operations vary depending on the type of data used in your project.

You don't have much experience with DSP? No problem, Edge Impulse usually uses a star to indicate the most recommended processing block based on your input data as shown in the image below.

In the case where the available processing blocks aren't suitable for your application, you can build your own custom processing blocks and import into your project.

Learning blocks

After adding your processing block, it is now time to add a learning block to make your impulse complete. A learning block is simply a neural network that is trained to learn on your data.

Learning blocks vary depending on what you want your model to do. It can be classification, regression, anomaly detection, image transfer learning or object detection. It can also be a custom transfer learning block (enterprise feature)

Learning blocks available with time-series projects:

Learning blocks available with image projects:

Learning blocks available with object detection projects:

The data sources page is actually much more than just adding data from external sources. It let you create complete automated data pipelines so you can work on your active learning strategies.

From there, you can import datasets from existing cloud storage buckets, automate and schedule the imports, and, trigger actions such as explore and label your new data, retrain your model, automatically build a new deployment task and more.

Add a data source

Click in + Add new data source and select where your data lives:

Click on Next, provide credentials:

Click on Verify credentials:

Here, you have several options to automatically label your data:

• Infer from folder name

In the example above, the structure of the folder is the following:

.
├── cars
│   ├── cars.01741.jpg
│   ├── cars.01743.jpg
│   ├── cars.01745.jpg
│   ├── ... (400 items)
├── unknown
│   ├── unknown.test_2547.jpg
│   ├── unknown.test_2548.jpg
│   ├── unknown.test_2549.jpg
│   ├── ... (400 items)
└── unlabeled
    ├── cars.02066.jpg
    ├── cars.02067.jpg
    ├── cars.02068.jpg
    └── ... (14 items)

3 directories, 814 files

The labels will be picked from the folder name and will be split between your training and testing set using the following ratio 80/20.

Note that the samples present in an unlabeled/ folder will be kept unlabeled in Edge Impulse Studio.

Alternatively, you can also organize your folder using the following structure to automatically split your dataset between training and testing sets:

.
├── testing
│   ├── cars
│   │   ├── cars.00012.jpg
│   │   ├── cars.00031.jpg
│   │   ├── cars.00035.jpg
│   │   └── ... (~150 items)
│   └── unknown
│       ├── unknown.test_1012.jpg
│       ├── unknown.test_1026.jpg
│       ├── unknown.test_1027.jpg
│       ├── ... (~150 items)
├── training
│   ├── cars
│   │   ├── cars.00006.jpg
│   │   ├── cars.00025.jpg
│   │   ├── cars.00065.jpg
│   │   └── ... (~600 items)
│   └── unknown
│       ├── unknown.test_1002.jpg
│       ├── unknown.test_1005.jpg
│       └── unknown.test_46.jpg
│       └── ... (~600 items)
└── unlabeled
    ├── cars.02066.jpg
    ├── cars.02067.jpg
    ├── cars.02068.jpg
    └── ... (14 items)

7 directories, 1512 files

• Infer from file name:

When using this option, only the file name is taken into account. The part before the first . will be used to set the label. E.g. cars.01741.jpg will set the label to cars.

• Keep the data unlabeled:

All the data samples will be unlabeled, you will need to label them manually before using them.

Finally, click on Next, post-sync actions.

From this view, you can automate several actions:

• Recreate data explorer
• Retrain model

  If needed, will retrain your model with the same impulse. If you enable this you'll also get an email with the new validation and test set accuracy.

  Note: You will need to have at least setup up and trained your project once.

• Create new version

  Store all data, configuration, intermediate results and final models.

• Create new deployment

  Builds a new library or binary with your updated model. Requires 'Retrain model' to also be enabled.

Run the pipeline

Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.

Run the pipeline from the UI

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.

Run the pipeline from code

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.

Note that you will need to create an API key to run the pipeline from code.

Schedule your pipeline jobs

By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.

Note that free users can only run the pipeline every 4 hours. If you are an enterprise customer, you can run this pipeline up to every minute.

Once the pipeline has successfully finish, you will receive an email like the following:

Webhooks

Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:

{
    "organizationId":XX,
    "pipelineId":XX,
    "pipelineName":"Import data from portal \"Data sources demo\"",
    "projectId":XXXXX,
    "success":true,
    "newItems":0,
    "newChecklistOK":0,
    "newChecklistFail":0
}

Edit your pipeline

As of today, if you want to update your pipeline, you need to edit the configuration json available in ⋮ -> Run pipeline from code.

Here is an example of what you can get if all the actions have been selected:

[
    {
        "name": "Fetch data from s3://data-pipeline/data-pipeline-example/infer-from-folder/",
        "builtinTransformationBlock": {
            "type": "s3-to-project",
            "endpoint": "https://s3.your-endpoint.com",
            "path": "s3://data-pipeline/data-pipeline-example/infer-from-folder/",
            "region": "fr-par",
            "accessKey": "XXXXX",
            "category": "split",
            "labelStrategy": "infer-from-folder-name",
            "secretKeyEncrypted": "xxxxxx"
        }
    },
    {
        "name": "Refresh data explorer",
        "builtinTransformationBlock": {
            "type": "project-action",
            "refreshDataExplorer": true
        }
    },
    {
        "name": "Retrain model",
        "builtinTransformationBlock": {
            "type": "project-action",
            "retrainModel": true
        }
    },
    {
        "name": "Create new version",
        "builtinTransformationBlock": {
            "type": "project-action",
            "createVersion": true
        }
    },
    {
        "name": "Create on-device deployment (C++ library)",
        "builtinTransformationBlock": {
            "type": "project-action",
            "buildBinary": "zip",
            "buildBinaryModelType": "int8"
        }
    }
]

Free projects have only access to the above builtinTransformationBlock.

Select "Copy as pipeline step" and paste it to the configuration json file.

There is a wide variety of devices that you can connect to your Edge Impulse project. These devices can help you collect datasets for your project, test your trained ML model and even deploy your ML model directly to your development board with a pre-built binary application (for ).

On the Devices tab, you'll find a list of all your connected devices and a guide on how to connect new devices that are currently supported by Edge Impulse.

To connect a new device, click on the Connect a new device button on the top right of your screen.

You will get a pop-up with multiple options of devices you can connect to your Edge Impulse project. Available options include:

The API references for the ingestion service, remote management service, and the studio API; plus SDK documentation for the acquisition and inferencing libraries can be found in the .

Services

SDK documentation

• SDKs for Node.js, Python, Go and C++

You can upload an already existing dataset to your project directly through the Edge impulse Studio. The data should be in the Data Acquisition Format (CBOR, JSON, CSV), or as WAV, JPG or PNG files.

To upload data using the uploader, go to the Data acquisition page and click on the uploader button as shown in the image below:

When uploading your data, you can choose the category you want your data to fall in i.e training set, testing set or automatically split the dataset between training and testing set. You can also choose whether to infer labels from files name or enter a label of which the files should automatically fall in.

The Raw Data block generates windows from data samples without any specific signal processing. It is great for signals that have already been pre-processed and if you just need to feed your data into the Neural Network block.

GitHub repository containing all DSP block code: .

Raw data parameters

Scaling

• Scale axes: Multiplies each axis by this number. This can be used to normalize your data between 0 and 1.

How does the raw data block work?

The Raw Data block retrieves raw samples and applies the Scaling parameter.

The Spectrogram processing block extracts time and frequency features from a signal. It performs well on audio data for non-voice recognition use cases, or on any sensor data with continuous frequencies.

GitHub repository containing all DSP block code: .

Spectrogram parameters

Spectrogram

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Frequency bands: The FFT size

Normalization

• Noise floor (dB): signal lower than this level will be dropped

How does the spectrogram block work?

It first divides the window in multiple overlapping frames. The size and number of frames can be adjusted with the parameters Frame length and Frame stride. For example with a window of 1 second, frame length of 0.02s and stride of 0.01s, it will create 99 time frames.

Each time frame is then divided in frequency bins using an FFT (Fast Fourier Transform) and we compute its power spectrum. The number of frequency bins equals to the Frequency bands parameter divided by 2 plus 1. We recommend keeping the Frequency bands (a.k.a. FFT size) value as a power of 2 for performances purpose. Finally the Noise floor value is applied to the power spectrum.

The features generated by the Spectrogram block are equal to the number of generated time frames times the number of frequency bins.

The Image block is dedicated to computer vision applications. It normalizes image data, and optionally reduce the color depth.

GitHub repository containing all DSP block code: .

Image parameters

• Color depth: Color depth to use (RGB or grayscale)

How does the image block work?

The Image performs normalization, converting each pixel's channel of the image to a float value between 0 and 1. If Grayscale is selected, each pixel is converted to a single value following the (Y' component only).

The Flatten block performs statistical analysis on the signal. It is useful for slow-moving averages like temperature data, in combination with other blocks.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Flatten parameters

Scaling

• Scale axes: Multiplies axes by this number

Method

• Average: Calculates the average value for the window

• Minimum: Calculates the minimum value in the window

• Maximum: Calculates the maximum value in the window

• Root-mean square: Calculates the RMS value of the window

• Standard deviation: Calculates the standard deviation of the window

• Skewness: Calculates the skewness of the window

• Kurtosis: Calculates the kurtosis of the window

How does the flatten block work?

The Flatten block first rescales axes of the signal if value is different than 1. Then statistical analysis is performed on each window, computing between 1 and 7 features for each axis, depending on the number of selected methods.

The IMU Syntiant block rescales raw data to 8 bits values to match the NDP101 chip input requirements.

Parameters

Scaling

• Scale 16 bits to 8 bits: Scale data to 8-bits values in the [-1, 1] range, raw data is divided by 2G (2 * 9.80665). Using Edge Impulse official firmwares, this parameter should be enabled as raw data is not rescaled. If this parameter is disabled the data samples will not be rescaled, you should disable this parameter if your raw data samples are already normalized to the [-1, 1] range.

How does the IMU Syntiant block work?

The IMU Syntiant block retrieves raw samples and applies the Scale 16 bits to 8 bits parameter.

Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio):

• Raw Data

• Flatten

• Image

• Spectral features

• Spectrogram

• Audio MFE

• Audio MFCC

• Audio Syntiant

• IMU Syntiant

The source code of these blocks are available in the Edge Impulse processing blocks GitHub repository.

If you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing, follow our tutorial on Building custom processing blocks.

Similarly to the Spectrogram block, the Audio MFE processing block extracts time and frequency features from a signal. However it uses a non-linear scale in the frequency domain, called Mel-scale. It performs well on audio data, mostly for non-voice recognition use cases when sounds to be classified can be distinguished by human ear.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Audio MFE parameters

Mel-filterbank energy features

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number: The number of triangular filters applied to the spectrogram

• FFT length: The FFT size

• Low frequency: Lowest band edge of Mel-scale filterbanks

• High frequency: Highest band edge of Mel-scale filterbanks

Normalization

• Noise floor (dB): signal lower than this level will be dropped

How does the MFE block work?

The features' extractions is similar to the Spectrogram (Frame length, Frame stride, and FFT length parameters are the same) but it adds 2 extra steps.

After computing the spectrogram, triangular filters are applied on a Mel-scale to extract frequency bands. They are configured with parameters Filter number, Low frequency and High frequency to select the frequency band and the number of frequency features to be extracted. The Mel-scale is a perceptual scale of pitches judged by listeners to be equal in distance from one another. The idea is to extract more features (more filter banks) in the lower frequencies, and less in the high frequencies, thus it performs well on sounds that can be distinguished by human ear.

The last step is to perform a local mean normalization of the signal, applying the Noise floor value to the power spectrum.

After extracting meaningful features from the raw signal using signal processing, you can now train your model using a learning block. We provide a number of pre-defined learning blocks:

• .

• .

• .

• .

• .

• .

• (Enterprise feature).

For most of the learning blocks (except K-means Anomaly Detection), you can use the Switch to expert mode button to access the full Keras API for custom architectures, , and more.

The two most common image processing problems are image classification and object detection.

Image classification takes an image as an input and outputs what type of object is in the image. This technique works great, even on microcontrollers, as long as we only need to detect a single object in the image.

On the other hand, object detection takes an image and outputs information about the class and number of objects, position, (and, eventually, size) in the image.

Edge Impulse provides two different methods to perform object detection:

• Using MobileNetV2 SSD FPN

• Using FOMO

Solving regression problems is one of the most common applications for machine learning models, especially in supervised machine learning. Models are trained to understand the relationship between independent variables and an outcome or dependent variable. The model can then be leveraged to predict the outcome of new and unseen input data, or to fill a gap in missing data.

Prerequisites

Labelling

To build a regression model you collect data as usual, but rather than setting the label to a text value, you set it to a numeric value.

Processing blocks

You can use any of the built-in signal processing blocks to pre-process your vibration, audio or image data, or use custom processing blocks to extract novel features from other types of sensor data.

Train your regression block

You have full freedom in modifying your neural network architecture - whether visually or through writing Keras code.

• Number of training cycles: Each time the training algorithm makes one complete pass through all of the training data with back-propagation and updates the model's parameters as it goes, it is known as an epoch or training cycle.

• Learning rate: The learning rate controls how much the models internal parameters are updated during each step of the training process. Or you can also see it as how fast the neural network will learn. If the network overfits quickly, you can reduce the learning rate

• Validation set size: The percentage of your training set held apart for validation, a good default is 20%

• Auto-balance dataset Mix in more copies of data from classes that are uncommon. Might help make the model more robust against overfitting if you have little data for some classes.

Test your regression model

If you want to see the accuracy of your model across your test dataset, go to Model testing. You can adjust the Maximum error percentage by clicking on "⋮" button.

Additional resources

• Predict the Future with Regression Models

• Estimate Weight From a Photo Using Visual Regression in Edge Impulse

If you have selected the Classification learning block in the Create impulse page, a NN Classifier page will show up in the menu on the left. This page becomes available after you've extracted your features from your DSP block.

The basic idea is that a neural network classifier will take some input data, and output a probability score that indicates how likely it is that the input data belongs to a particular class.

So how does a neural network know what to predict? The neural network consists of a number of layers, each of which is made up of a number of neurons. The neurons in the first layer are connected to the neurons in the second layer, and so on. The weight of a connection between two neurons in a layer is randomly determined at the beginning of the training process. The neural network is then given a set of training data, which is a set of examples that it is supposed to predict. The network's output is compared to the correct answer and, based on the results, the weights of the connections between the neurons in the layer are adjusted. This process is repeated a number of times, until the network has learned to predict the correct answer for the training data.

A particular arrangement of layers is referred to as an architecture, and different architectures are useful for different tasks. This way, after a lot of iterations, the neural network learns; and will eventually become much better at predicting new data.

On this page, you can configure the model and the training process and, have an overview of your model performances.

Neural Network settings

• Number of training cycles: Each time the training algorithm makes one complete pass through all of the training data with back-propagation and updates the model's parameters as it goes, it is known as an epoch or training cycle.

• Learning rate: The learning rate controls how much the models internal parameters are updated during each step of the training process. Or you can also see it as how fast the neural network will learn. If the network overfits quickly, you can reduce the learning rate

• Validation set size: The percentage of your training set held apart for validation, a good default is 20%

• Auto-balance dataset Mix in more copies of data from classes that are uncommon. Might help make the model more robust against overfitting if you have little data for some classes.

Neural Network architecture

Depending on your project type, we may offer to choose between different architecture presets to help you get started.

The neural network architecture takes as inputs your extracted features, and pass the features to each layer of your architecture. In the classification case, the last used layer is a softmax layer. It is this last layer that gives the probability of belonging to one of the classes.

From the visual (simple) mode, you can add the following layers:

If have advanced knowledge in machine learning and Keras, you can switch to the Expert Mode and access the full Keras API to use custom architectures:

Training output

This panel displays the output logs during the training. The previous training logs can also be retrieved from the Jobs tab in the Dashboard page (enterprise feature).

Model performances

This section gives an overview of your model performances and helps you evaluate your model. It can help you determine if the model is capable of meeting your needs or if you need to test other hyper parameters and architectures.

From the Last training performances you can retrieve your validation accuracy and loss.

The Confusion matrix is one of most useful tool to evaluate a model. it tabulates all of the correct and incorrect responses a model produces given a set of data. The labels on the side correspond to the actual labels in each sample, and the labels on the top correspond to the predicted labels from the model.

The features explorer, like in the processing block views, indicated the spatial distribution of your input features. In this page, you can visualize which ones have been correctly classified and which ones have not.

On-device performance: Based on the target you chose in the Dashboard page, we will output estimations for the inferencing time, peak RAM usage and flash usage. This will help you validate that your model will be able to run on your device based on its constraints.

Neural networks are great, but they have one big flaw. They're terrible at dealing with data they have never seen before (like a new gesture). Neural networks cannot judge this, as they are only aware of the training data. If you give it something unlike anything it has seen before it'll still classify as one of the four classes.

K-means clustering

This method looks at the data points in a dataset and groups those that are similar into a predefined number K of clusters. A threshold value can be added to detect anomalies: if the distance between a data point and its nearest centroid is greater than the threshold value, then it is an anomaly.

The main difficulty resides in choosing K, since data in a time series is always changing and different values of K might be ideal at different times. Besides, in more complex scenarios where there are both local and global outliers, many outliers might pass under the radar and be assigned to a cluster.

Features importance (optional)

In most of your DSP blocks, you have an option to calculate the feature importance. Edge Impulse Studio will then output a Feature Importance graphic that will help you determine which axes and values generated from your DSP block are most significant to analyze when you want to do anomaly detection.

This process of generating features and determining the most important features of your data will further reduce the amount of signal analysis needed on the device with new and unseen data.

Setting up the anomaly detection block

In your anomaly detection block, you can click on the Select suggested axes button to harness the value of the feature importance output.

Here is the process in the background:

• Create X number of clusters and group all the data.

• For each of these clusters we store the center and the size of the cluster.

• During inference we calculate the closest cluster for a new data point, and show the distance from the edge of the cluster. If it’s within a cluster (no anomaly)you thus get a value below 0.

In the above picture, known clusters in are in blue, new classified data in orange. It's clearly outside of any known clusters and can thus be tagged as an anomaly.

Additional resources

• Tutorial: Continuous Motion Recognition

• Blog post: Advanced Anomaly Detection with Feature Importance

The Audio Syntiant processing block extracts time and frequency features from a signal. It is similar to the Audio MFE but performs additional processing specific to the Syntiant NDP101 chip. This block can be used only with Syntiant targets.

Audio Syntiant parameters

Log Mel-filterbank energy features

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number (fixed): The number of triangular filters applied to the spectrogram

• FFT length (fixed): The FFT size

• Low frequency (fixed): Lowest band edge of Mel-scale filterbanks

• High frequency (fixed): Highest band edge of Mel-scale filterbanks

• Coefficient: Pre-emphasis coefficient

How does the Syntiant block work?

The features' extractions is a proprietary algorithm from Syntiant. However parameters are very close to the Audio MFE. Pre-emphasis coefficient is applied first to amplify higher frequencies. The signal is then divided in overlapping frames, defined by the Frame length and Frame stride to extract speech features.

The Audio MFCC blocks extracts coefficients from an audio signal. Similarly to the Audio MFE block, it uses a non-linear scale called Mel-scale. It is the reference block for speech recognition and can also performs well on some non-human voice use cases.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Audio MFCC parameters

Mel Frequency Cepstral Coefficients

• Number of coefficients: Number of cepstral coefficients to keep after applying Discrete Cosine Transform

• Frame length: The length of each frame in seconds

• Frame stride: The step between successive frame in seconds

• Filter number: The number of triangular filters applied to the spectrogram

• FFT length: The FFT size

• Low frequency: Lowest band edge of Mel-scale filterbanks

• High frequency: Highest band edge of Mel-scale filterbanks

• Window size: The size of sliding window for local cepstral mean normalization. Windows size must be odd.

Pre-emphasis

• Coefficient: The pre-emphasizing coefficient to apply to the input signal (0 equals to no filtering)

• Note: Shift has been removed and set to 1 for all future projects. Older & existing projects can still change this value or use an existing value.

How does the MFCC block work?

The features' extractions adds one extra step to the MFE block resulting in a compressed representation of the filterbanks. A Discrete Cosine Transform is applied on each filterbank to extract cepstral coefficients. 13 coefficients are usually retained, the rest are discarded as they represent fast changes not useful for speech recognition.

Training and deploying high performing ML models is usually considered as a continuous process rather than a one time exercise. When you are validating your model and discover an overfit, you might consider adding some more diverse data then perform model retraining while maintaining the initially set DSP and Neural Network block configurations.

Also during inference If you find that the data distribution has drifted significantly from the initial training distribution, it is usually a good common practice to retrain your model on the newer data distribution to keep up with the high model performance.

The “Retrain Model” feature in the Edge Impulse Studio is usually useful when adding new data to your project. It uses known parameters from your selected DSP and ML blocks then uses them to automatically regenerate new features and retrain the Neural Network model in one single step. You can consider this a shortcut for retraining your model since you don’t need to go through all the blocks in your impulse one by one again.

To retrain your model after adding some data, navigate to "Retrain Model" on the studio and click "Train model"

It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.

How to get started?

To build your first object detection models using MobileNetV2 SSD FPN-Lite :

1. Create a new project in Edge Impulse.

2. Make sure to set your labelling method to 'Bounding boxes (object detection)'.

3. Collect and prepare your dataset as in Object detection

4. Resize your image to fit 320x320px

5. Add an 'Object Detection (Images)' block to your impulse.

6. Under Images, choose RGB.

7. Under Object detection, select 'Choose a different model' and select 'MobileNetV2 SSD FPN-Lite 320x320'

8. You can start your training with a learning rate of '0.15'

1. Click on 'Start training'

MobileNetV2 SSD FPN-Lite 320x320 is available with Edge Impulse for Linux

How does this 🪄 work?

Here, we are using the MobileNetV2 SSD FPN-Lite 320x320 pre-trained model. The model has been trained on the COCO 2017 dataset with images scaled to 320x320 resolution.

In the MobileNetV2 SSD FPN-Lite, we have a base network (MobileNetV2), a detection network (Single Shot Detector or SSD) and a feature extractor (FPN-Lite).

Base network:

MobileNet, like VGG-Net, LeNet, AlexNet, and all others, are based on neural networks. The base network provides high-level features for classification or detection. If you use a fully connected layer and a softmax layer at the end of these networks, you have a classification.

But you can remove the fully connected and the softmax layers, and replace it with detection networks, like SSD, Faster R-CNN, and others to perform object detection.

Detection network:

The most common detection networks are SSD (Single Shot Detection) and RPN (Regional Proposal Network).

When using SSD, we only need to take one single shot to detect multiple objects within the image. On the other hand, regional proposal networks (RPN) based approaches, such as R-CNN series, need two shots, one for generating region proposals, one for detecting the object of each proposal.

As a consequence, SSD is much faster compared with RPN-based approaches but often trades accuracy with real-time processing speed. They also tend to have issues in detecting objects that are too close or too small.

Feature Pyramid Network:

Detecting objects in different scales is challenging in particular for small objects. Feature Pyramid Network (FPN) is a feature extractor designed with feature pyramid concept to improve accuracy and speed.

Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio), but they might not be suitable for all applications. Perhaps you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing. In this tutorial you'll learn how to support these use cases by adding custom processing blocks to the studio.

Prerequisites

Make sure you followed the Continuous motion recognition tutorial, and have a trained impulse.

1. Building your first custom processing block

Processing blocks take data and configuration parameters in, and return features and visualizations like graphs or images. To communicate to custom processing blocks, Edge Impulse studio will make HTTP calls to the block, and then use the response both in the UI, while generating features, or when training a machine learning model. Thus, to load a custom processing block we'll need to run a small server that responds to these HTTP calls. You can write this in any language, but we have created an example in Python. To load this example, open a terminal and run:

$ git clone https://github.com/edgeimpulse/example-custom-processing-block-python

This creates a copy of the example project locally. Then, you can run the example either through Docker or locally via:

Docker

$ docker build -t custom-blocks-demo .
$ docker run -p 4446:4446 -it --rm custom-blocks-demo

Locally

$ pip3 install -r requirements.txt
$ python3 dsp-server.py

Then go to http://localhost:4446 and you should be shown some information about the block.

Exposing the processing block to the world

As this block is running locally the studio cannot reach the block. To resolve this we can use ngrok which can make a local port accessible from a public URL. After you've finished development you can move the processing block to a server with a publicly accessible address (or run it on our infrastructure through your enterprise account). To set up a tunnel:

1. Sign up for ngrok.

2. Install the ngrok binary for your platform.

3. Get a URL to access the processing block from the outside world via:

$ ngrok http 4446
# or
$ ./ngrok http 4446

This yields a public URL for your block under Forwarding. Note down the address that includes https://.

Session Status                online
Account                       Edge Impulse (Plan: Free)
Version                       2.3.35
Region                        United States (us)
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://4d48dca5.ngrok.io -> http://localhost:4446
Forwarding                    https://4d48dca5.ngrok.io -> http://localhost:4446

Adding the custom block to Edge Impulse

Now that the custom processing block was created, and you've made it accessible to the outside world, you can add this block to Edge Impulse. In a project, go to Create Impulse, click Add a processing block, choose Add custom block (in the bottom left corner of the modal), and paste in the public URL of the block:

After you click Add block the block will show like any other processing block.

Add a learning bloc, then click Save impulse to store the impulse.

2. Adding configuration options

Processing blocks have configuration options which are rendered on the block parameter page. These could be filter configurations, scaling options, or control which visualizations are loaded. These options are defined in the parameters.json file. Let's add an option to smooth raw data. Open example-custom-processing-block-python/parameters.json and add a new section under parameters:

        {
            "group": "Filter",
            "items": [
                {
                    "name": "Smooth",
                    "value": false,
                    "type": "boolean",
                    "help": "Whether to smooth the data",
                    "param": "smooth"
                }
            ]
        }

Then, open example-custom-processing-block-python/dsp.py and replace its contents with:

import numpy as np

def generate_features(draw_graphs, raw_data, axes, sampling_freq, scale_axes, smooth):
    return { 'features': raw_data * scale_axes, 'graphs': [] }

Restart the Python script, and then click Custom block in the studio (in the navigation bar). You now have a new option 'Smooth'. Every time an option changes we'll re-run the block, but as we have not written any code to respond to these changes nothing will happen.

2.1 Valid configuration types

We support a number of different types for configuration fields. These are:

• int - renders a numeric textbox that expects integers.

• float - renders a numeric textbox that expects floating point numbers.

• string - renders a textbox that expects a string.

• boolean - renders a checkbox.

• select - renders a dropdown box. This also requires the parameter valid which should be an array of valid values. E.g. this renders a dropdown box with options 'low', 'high' and 'none':

{
    "name": "Type",
    "value": "low",
    "help": "Type of filter to apply to the raw data",
    "type": "select",
    "valid": [ "low", "high", "none" ],
    "param": "filter-type"
}

3. Implementing smoothing and drawing graphs

To show the user what is happening we can also draw visuals in the processing block. Right now we support graphs (linear and logarithmic) and arbitrary images. By showing a graph of the smoothed sample we can quickly identify what effect the smooth option has on the raw signal. Open dsp.py and replace the content with the following script. It contains a very basic smoothing algorithm and draws a graph:

import numpy as np

def smoothing(y, box_pts):
    box = np.ones(box_pts) / box_pts
    y_smooth = np.convolve(y, box, mode='same')
    return y_smooth

def generate_features(draw_graphs, raw_data, axes, sampling_freq, scale_axes, smooth):
    # features is a 1D array, reshape so we have a matrix with one raw per axis
    raw_data = raw_data.reshape(int(len(raw_data) / len(axes)), len(axes))

    features = []
    smoothed_graph = {}

    # split out the data from all axes
    for ax in range(0, len(axes)):
        X = []
        for ix in range(0, raw_data.shape[0]):
            X.append(raw_data[ix][ax])

        # X now contains only the current axis
        fx = np.array(X)

        # first scale the values
        fx = fx * scale_axes

        # if smoothing is enabled, do that
        if (smooth):
            fx = smoothing(fx, 5)

        # we save bandwidth by only drawing graphs when needed
        if (draw_graphs):
            smoothed_graph[axes[ax]] = list(fx)

        # we need to return a 1D array again, so flatten here again
        for f in fx:
            features.append(f)

    # draw the graph with time in the window on the Y axis, and the values on the X axes
    # note that the 'suggestedYMin/suggestedYMax' names are incorrect, they describe
    # the min/max of the X axis
    graphs = []
    if (draw_graphs):
        graphs.append({
            'name': 'Smoothed',
            'X': smoothed_graph,
            'y': np.linspace(0.0, raw_data.shape[0] * (1 / sampling_freq) * 1000, raw_data.shape[0] + 1).tolist(),
            'suggestedYMin': -20,
            'suggestedYMax': 20
        })

    return { 'features': features, 'graphs': graphs }

Restart the script, and click the Smooth toggle to observe the difference. Congratulations! You have just created your first custom processing block.

3.1 Adding features to labels

If you extract set features from the signal, like the mean, that you that return, you can also label these features. These labels will be used in the feature explorer. To do so, add a labels array that contains strings that map back to the features you return (labels and features should have the same length).

4. Other type of graphs

In the previous step we drew a linear graph, but you can also draw logarithmic graphs or even full images. This is done through the type parameter:

4.1 Logarithmic graphs

This draws a graph with a logarithmic scale:

    graphs.append({
        'name': 'Logarithmic example',
        'X': {
            'Axis title': [ pow(10, i) for i in range(10) ]
        },
        'y': np.linspace(0, 10, 10).tolist(),
        'suggestedXMin': 0,
        'suggestedXMax': 10,
        'suggestedYMin': 0,
        'suggestedYMax': 1e+10,
        'type': 'logarithmic'
    })

4.2 Images

To show an image you should return the base64 encoded image and its MIME type. Here's how you draw a small PNG image:

    from PIL import Image, ImageDraw, ImageFont, ImageFilter

    # create a new image, and draw some text on it
    im = Image.new ('RGB', (438, 146), (248, 86, 44))
    draw = ImageDraw.Draw(im)
    draw.text((10, 10), 'Hello world!', fill=(255, 255, 255))

    # save the image to a buffer, and base64 encode the buffer
    with io.BytesIO() as buf:
        im.save(buf, format='png', bbox_inches='tight', pad_inches=0)
        buf.seek(0)
        image = (base64.b64encode(buf.getvalue()).decode('ascii'))

        # append as a new graph
        graphs.append({
            'name': 'Image from custom block',
            'image': image,
            'imageMimeType': 'image/png',
            'type': 'image'
        })

4.3 Dimensionality reduction

If you output high-dimensional data (like a spectrogram or an image) you can enable dimensionality reduction for the feature explorer. This will run UMAP over the data to compress the features into three dimensions. To do so, set:

"visualization": "dimensionalityReduction"

On the info object in parameters.json.

4.4 Full documentation

For all options that you can return in a graph, see the Run DSP return types in the API documentation.

5. Running on device

Your custom block behaves exactly the same as any of the built-in blocks. You can process all your data, train neural networks or anomaly blocks, and validate that your model works. However, we cannot automatically generate optimized native code for the block, like we do for built-in processing blocks, but we try to help you write this code as much as possible. When you export your project to a C++ library we generate struct's for all the configuration options, and you only need to implement the extract_custom_block_features function (you can change this name through the cppType parameter in parameters.json).

An example of this function for the spectral analysis block is listed in the inferencing sdk.

6. Other resources

Blog post: Utilize Custom Processing Blocks in Your Image ML Pipelines

7. Conclusion

With good feature extraction you can make your machine learning models smaller and more reliable, which are both very important when you want to deploy your model on embedded devices. With custom processing blocks you can now develop new feature extraction pipelines straight from Edge Impulse. Whether you're following the latest research, want to implement proprietary algorithms, or are just exploring data.

For inspiration we have published all our own blocks here: edgeimpulse/processing-blocks. If you've made an interesting block that you think is valuable for the community, please let us know on the forums or by opening a pull request. We'd be happy to help write efficient native code for the block, and then publish it as a standard block!

Your Edge Impulse organization helps your team with the full lifecycle of your TinyML deployment. It contains tools to collect and maintain large datasets, allows your data scientists to quickly access relevant data through their familiar tools, adds versioning and traceability to your machine learning models, and lets you quickly create new Edge Impulse projects for on-device deployment.

To get started, follow these tutorials:

• Collaborating on projects - to work with your colleagues on one project.

• Building your first dataset - to build up a shared dataset for your organization.

• Upload portals - to allow external parties to securely contribute data to your datasets.

• Creating a transformation block - to quickly extract features from your dataset.

• Building deployment blocks - to create custom deployment targets for your products.

• Hosting custom DSP blocks - to create and host your custom signal processing techniques and use it directly in your projects.

• Adding custom transfer learning models - to use your custom neural networks architectures and load pre-trained weights.

When creating an impulse to solve an image classification problem, you will most likely want to use 'transfer learning' as learning block. This is particularly true especially when working with a relatively small dataset.

Transfer learning is the process of taking features learned from one problem and leveraging it on a new but related problem. Most of the time these features are learned from large scale datasets with common objects hence making it faster & more accurate to tune and adapt to new tasks.

To choose transfer learning as your learning block, go to create impulse and click on 'Add a Learning Block', and select 'Transfer Learning'

To choose your preferred pretrained network, go to Transfer learning on the left side of your screen and click 'choose a different model'. A pop up will appear on your screen with a list of models to choose from as shown in the image below.

Edge Impulse uses state of the art MobileNetV1 & V2 architectures trained on ImageNet dataset as it's pretrained network for you to fine-tune for your specific application. The pretrained networks comes with varying input blocks ranging from 96x96 to 320x320 and both RGB & Grayscale images for you to choose from depending on your application & target deployment hardware.

Neural Network Settings

Before you start training your model, you need to set the following neural network configurations:

• Number of training cycles: Each time the training algorithm makes one complete pass through all of the training data with back-propagation and updates the model's parameters as it goes, it is known as an epoch or training cycle.

• Learning rate: The learning rate controls how much the models internal parameters are updated during each step of the training process. Or you can also see it as how fast the neural network will learn. If the network overfits quickly, you can reduce the learning rate

• Validation set size: The percentage of your training set held apart for validation, a good default is 20%.

You might also need to enable auto balance to prevent model bias or even enable data augmentation to increase the size of your dataset and have more diverse dataset to prevent overfitting.

The preset configurations just don't work for your model? No worries, Expert Mode is for you! The Expert mode gives you full control of your model so that you can configure it however you want. To enable the expert mode, just click on the "⋮" button and toggle the expert mode.

You can use the expert mode to change your loss function, optimizer, print your model architecture and even set an early stopping callback to prevent overfitting your model.

Live classification lets you validate your model with data captured directly from any device or supported development board. This gives you a picture on how your model will perform with real world data. To achieve this, go to Live classification and connect the device or development board you want to capture data from.

Using a fully supported development board

All of your connected devices and sensors will appear under Devices as shown below. The devices can be connected through the Edge Impulse CLI or WebUSB:

Using your mobile phone

To perform live classification using your phone, go to Devices and click Connect a new device then select "Use your mobile phone". Scan the QR code using your phone then click Switch to classification mode and start sampling.

Using your computer

To perform live classification using your computer, go to Devices and click Connect a new device then select "Use your computer". Give permissions on your computer then click Switch to classification mode and start sampling.

When collecting data, we split the dataset into training and testing sets. The model was trained with only the training set, and the testing set is used to validate how well the model will perform on un-seen data. This will ensure that the model has not learned to overfit the training data, which is a common occurrence.

To test your model, go to Model testing, and click Test all. The model will classify all of the test set samples and give you an overall accuracy of how your model performed.

This is also accompanied by a confusion matrix to show you how your models performs for each class.

Evaluating individual samples

To see a classification in detail, go to the individual sample you are want to evaluate and click the three dots next to it, then just select show classification. This will open a new window that will display the expected outcome, and the predicted output of your model with its accuracy. This detailed view can also give you a hint on why an item has been misclassified.

Setting confidence threshold

Every learning block has a threshold. This can be the minimum confidence that a neural network needs to have, or the maximum anomaly score before a sample is tagged as an anomaly. You can configure these thresholds to tweak the sensitivity of these learning blocks. This affects both live classification and model testing.

The Spectral features block extracts frequency and power characteristics of a signal. Low-pass and high-pass filters can also be applied to filter out unwanted frequencies. It is great for analyzing repetitive patterns in a signal, such as movements or vibrations from an accelerometer.

GitHub repository containing all DSP block code: edgeimpulse/processing-blocks.

Spectral analysis parameters

Scaling

• Scale axes: Multiplies axes by this number to scale data from the sensor

Filter

• Type: Type of filter to apply to the raw data (low-pass, high-pass, or none)

• Cut-off frequency: Cut-off frequency of the filter in hertz

• Order: Order of the Butterworth filter

Spectral power

• FFT length: The FFT size

• No. of peaks: Number of spectral power peaks to extract

• Peaks threshold: Minimum threshold to extract a peak (frequency domain normalized to [0, 1])

• Power edges: Splits the power spectral density in various buckets (V**2/Hz unit)

How does the spectral features block work?

The spectral analysis block generates 3 types of features per axis:

• The root mean square of the filter output (1 scalar value)

• The frequency and height of spectral power peaks

• The average power spectral density for each bucket

The raw signal is first scaled up or down based on the Scale axes value and offset by its mean value. A Butterworth filter is then applied (except if None selected); the order of the filter indicates how steep the slope is at the cut-off frequency.

At this point the root mean square of the filter output is added to the features' list.

The filter output is then used to extract:

• Spectral power peaks: after performing the FFT, the No. of peaks peaks with the highest magnitude are stored in the features' list (frequency and height). Peaks threshold can be tuned to define a minimum value to extract a peak.

• PSD for each bucket: after computing the power spectral density, the signal is divided in power buckets, defined by the Power edges parameter. Each sample frequency of the PSD is added to a power bucket. The power buckets are then averaged and added to the features' list. This process allows to extract a power representation of the signal.

Number of generated features

Let's consider an input signal with 3 axis and the following parameters:

• No. of peaks = 3

• Power edges = 0.1, 0.5, 1.0, 2.0, 5.0

The number of generated features per axis is:

• 1 value for RMS of the filter output

• 6 values for power peaks (frequency & height for each peak)

• 4 values for power buckets (number of Power edges - 1)

33 features are generated in total for the input signal.

Edge Impulse FOMO (Faster Objects, More Objects) is a novel machine learning algorithm that brings object detection to highly constrained devices. It lets you count objects, find the location of objects in an image, and track multiple objects in real-time using up to 30x less processing power and memory than MobileNet SSD or YOLOv5.

For example, FOMO lets you do 60 fps. object detection on a Raspberry Pi 4:

And here's FOMO doing 30 fps. object detection on an Arduino Nicla Vision (Cortex-M7 MCU), using 245K RAM.

You can find the complete Edge Impulse project with the beers vs. cans model, including all data and configuration here: https://studio.edgeimpulse.com/public/89078/latest.

How does this 🪄 work?

So how does that work? First, a small primer. Let's say you want to detect whether you see a face in front of your sensor. You can approach this in two ways. You can train a simple binary classifier, which says either "face" or "no face", or you can train a complex object detection model which tells you "I see a face at this x, y point and of this size". Object detection is thus great when you need to know the exact location of something, or if you want to count multiple things (the simple classifier cannot do that) - but it's computationally much more intensive, and you typically need much more data for it.

The design goal for FOMO was to get the best of both worlds: the computational power required for simple image classification, but with the additional information on location and object count that object detection gives us.

Heat maps

The first thing to realize is that while the output of the image classifier is "face" / "no face" (and thus no locality is preserved in the outcome) the underlying neural network architecture consists of a number of convolutional layers. A way to think about these layers is that every layer creates a diffused lower-resolution image of the previous layer. E.g. if you have a 16x16 image the width/height of the layers may be:

1. 16x16

2. 4x4

3. 1x1

Each 'pixel' in the second layer maps roughly to a 4x4 block of pixels in the input layer, and the interesting part is that locality is somewhat preserved. The 'pixel' in layer 2 at (0, 0) will roughly map back to the top left corner of the input image. The deeper you go in a normal image classification network, the less of this locality (or "receptive field") is preserved until you finally have just 1 outcome.

FOMO uses the same architecture, but cuts off the last layers of a standard image classification model and replaces this layer with a per-region class probability map (e.g. a 4x4 map in the example above). It then has a custom loss function which forces the network to fully preserve the locality in the final layer. This essentially gives you a heatmap of where the objects are.

The resolution of the heat map is determined by where you cut off the layers of the network. For the FOMO model trained above (on the beer bottles) we do this when the size of the heat map is 8x smaller than the input image (input image of 160x160 will yield a 20x20 heat map), but this is configurable. When you set this to 1:1 this actually gives you pixel-level segmentation and the ability to count a lot of small objects.

Training on centroids

A difference between FOMO and other object detection algorithms is that it does not output bounding boxes, but it's easy to go from heat map to bounding boxes. Just draw a box around a highlighted area.

However, when working with early customers we realized that bounding boxes are merely an implementation detail of other object detection networks, and are not a typical requirement. Very often the size of objects is not important as cameras are in fixed locations (and objects thus fixed size), but rather you just want the location and the count of objects.

Thus, we now train on the centroids of objects. This makes it much easier to count objects that are close (every activation in the heat map is an object), and the convolutional nature of the neural network ensures we look around the centroid for the object anyway.

A downside of the heat map is that FOMO is that each cell acts as its own classifier. E.g. if your classes are "lamp", "plant" and "background" each cell will be either lamp, plant, or background. It's thus not possible to detect objects with overlapping centroids. You can see this in the Raspberry Pi 4 video above at 00:18 where the beer bottles are too close together. This can be solved by using a higher resolution heat map.

Flexible and very, very fast

A really cool benefit of FOMO is that it's fully convolutional. If you set an image:heat map factor of 8 you can throw in a 96x96 image (outputs 12x12 heat map), a 320x320 image (outputs 40x40 heat map), or even a 1024x1024 image (outputs 128x128 heat map). This makes FOMO incredibly flexible, and useful even if you have very large images that need to be analyzed (e.g. in fault detection where the faults might be very, very small). You can even train on smaller patches, and then scale up during inference.

Additionally FOMO is compatible with any MobileNetV2 model. Depending on where the model needs to run you can pick a model with a higher or lower alpha, and transfer learning also works (although you need to train your base models specifically with FOMO in mind). This makes it easy for end customers to use their existing models and fine-tune them with FOMO to also add locality (e.g. we have customers with large transfer learning models for wildlife detection).

Together this gives FOMO the capabilities to scale from the smallest microcontrollers all the way to full gateways or GPUs. Just some numbers:

1. The video on the top classifies 60 times / second on a stock Raspberry Pi 4 (160x160 grayscale input, MobileNetV2 0.1 alpha). This is 20x faster than MobileNet SSD which does ~3 frames/second.

2. The second video on the top classifies 30 times / second on an Arduino Nicla Vision board with a Cortex-M7 MCU running at 480MHz) in ~240K of RAM (96x96 grayscale input, MobileNetV2 0.35 alpha).

3. During Edge Impulse Imagine we demonstrated a FOMO model running on a Himax WE-I Plus doing 14 frames per second on a DSP (video). This model ran in under 150KB of RAM (96x96 grayscale input, MobileNetV2 0.1 alpha). [1]

4. The smallest version of FOMO (96x96 grayscale input, MobileNetV2 0.05 alpha) runs in <100KB RAM and ~10 fps. on a Cortex-M4F at 80MHz. [1]

[1] Models compiled using EON Compiler.

How to get started?

To build your first FOMO models:

1. Create a new project in Edge Impulse.

2. Make sure to set your labeling method to 'Bounding boxes (object detection)'.

3. Collect and prepare your dataset as in Object detection

4. Add an 'Object Detection (Images)' block to your impulse.

5. Under Images, select 'Grayscale'

6. Under Object detection, select 'Choose a different model' and select one of the FOMO models.

7. Make sure to lower the learning rate to 0.001 to start.

FOMO is currently compatible with all fully-supported development boards that have a camera, and with Edge Impulse for Linux (any client). Of course, you can export your model as a C++ Library and integrate it as usual on any device or development board, the output format of models is compatible with normal object detection models; and our SDK runs on almost anything under the sun (see Running your impulse locally for an overview) from RTOS's to bare-metal to special accelerators and GPUs.

Expert mode tips

Additional configuration for FOMO can be accessed via expert mode.

Object weighting

FOMO is sensitive to the ratio of objects to background cells in the labelled data. By default the configuration is to weight object output cells x100 in the loss function, object_weight=100, as a way of balancing what is usually a majority of background. This value was chosen as a sweet spot for a number of example use cases. In scenarios where the objects to detect are relatively rare this value can be increased, e.g. to 1000, to have the model focus even more on object detection (at the expense of potentially more false detections).

MobileNet cut point

FOMO uses MobileNetV2 as a base model for its trunk and by default does a spatial reduction of 1/8th from input to output (e.g. a 96x96 input results in a 12x12 output). This is implemented by cutting MobileNet off at the intermediate layer block_6_expand_relu

Choosing a different cut_point results in a different spatial reduction; e.g. if we cut higher at block_3_expand_relu FOMO will instead only do a spatial reduction of 1/4 (i.e. a 96x96 input results in a 24x24output)

Note though; this means taking much less of the MobileNet backbone and results in a model with only 1/2 the params. Switching to a higher alpha may counteract this parameter reduction. Later FOMO releases will counter this parameter reduction with a UNet style architecture.

FOMO classifier capacity

FOMO can be thought of logically as the first section of MobileNetV2 followed by a standard classifier where the classifier is applied in a fully convolutional fashion.

In the default configuration this FOMO classifier is equivalent to a single dense layer with 32 nodes followed by a classifier with num_classes outputs.

For a three way classifier, using the default cut point, the result is a classifier head with ~3200 parameters.

 LAYER                          SHAPE                NUMBER OF PARAMETERS
 block_6_expand_relu (ReLU)     (None, 20, 20, 96)   0                                         
 head (Conv2D)                  (None, 20, 20, 32)   3104                                            
 logits (Conv2D)                (None, 20, 20, 3)    99

We have the option of increasing the capacity of this classifier head by either 1) increasing the number of filters in the Conv2D layer, 2) adding additional layers or 3) doing both.

For example we might change the number of filters from 32 to 16, as well as adding another convolutional layer, as follows.

 LAYER                          SHAPE                NUMBER OF PARAMETERS
 block_6_expand_relu (ReLU)     (None, 20, 20, 96)   0
 head_1 (Conv2D)                (None, 20, 20, 16)   1552                                         
 head_2 (Conv2D)                (None, 20, 20, 16)   272                                          
 logits (Conv2D)                (None, 20, 20, 3)    51

For some problems an additional layer can improve performance, and in this case actually uses less parameters. It can though potentially take longer to train and require more data. In future releases the tuning of this aspect of FOMO can be handled by the EON Tuner.

The EON Tuner helps you find and select the best embedded machine learning model for your application within the constraints of your target device. The EON Tuner analyzes your input data, potential signal processing blocks, and neural network architectures - and gives you an overview of possible model architectures that will fit your chosen device's latency and memory requirements.

Getting Started

First, make sure you have an audio, motion, or image classification project in your Edge Impulse account to run the EON Tuner with. No projects yet? Follow one of our tutorials to get started:

• Adding sight to your sensors.

• Building a continuous motion recognition system.

• Recognizing sounds from audio.

• Responding to your voice.

1. Log in to the Edge Impulse Studio and open a project.
2. Select the EON Tuner tab.
3. Click the Configure target button to select your model’s dataset category, target device, and time per inference (in ms).
4. Click on the Dataset category dropdown and select the use case unique to your motion, audio, or image classification project.
5. Click Save and then select Start EON Tuner
6. Wait for the EON Tuner to finish running, then click the Select button next to your preferred DSP/neural network model architecture to save as your project’s primary blocks:
7. Click on the DSP and Neural Network tabs within your Edge Impulse project to see the parameters the EON Tuner has generated and selected for your dataset, use case, and target device hardware
8. Now you’re ready to deploy your automatically configured Edge Impulse model to your target edge device!

Features

The EON Tuner performs end-to-end optimizations, from the digital signal processing (DSP) algorithm to the machine learning model, helping you find the ideal trade-off between these two blocks to achieve optimal performance on your target hardware. The unique features and options available in the EON Tuner are described below.

Targets

The Tuner can directly analyze the performance on any device fully supported by Edge Impulse. If you are targeting a different device, select a similar class of processor or leave the target as the default. You'll have the opportunity to further refine the EON tuner results to fit your specific target and application later.

Dataset Categories

The EON Tuner currently supports three different types of sensor data: motion, images, and audio. From these, the tuner can optimize for different types of common applications or dataset categories.

Input

The EON Tuner evaluates different configurations for creating samples from your dataset. For time series data, the tuner tests different sample window sizes and increment amounts. For image data, the tuner compares different image resolutions.

Processing Blocks

Depending on the selected dataset category, the EON Tuner considers a variety of Processing blocks when evaluating model architectures. The EON Tuner will test different parameters and configurations of these processing blocks.

Learning Blocks

Different model architectures, hyper-parameters, and even data augmentation techniques are evaluated by the EON Tuner. The tuner combines these different neural networks with the processing and input options described above, and then compares the end-to-end performance

Tuner Operation and Results

During operation, the tuner first generates many variations of input, processing, and learning blocks. It then schedules training and testing of each variation. The top level progress bar shows tests started (blue stripes) as well completed tests (solid blue), relative to the total number of generated variations.

Detailed logs of the run are also available. To view them, click on the button next to Target shown below.

As results become available, they will appear in the tuner window. Each result shows the on-device performance and accuracy, as well as details on the input, processing, and learning blocks used. Clicking Select sets a result as your project's primary impulse, and from there you can view or modify the design in the Impulse Design tabs.

Filters

While the EON Tuner is running, you can filter results by job status, processing block, and learning block categories.

Views

View options control what information is shown in the tuner results. You can choose which dataset is used when displaying model accuracy, as well as whether to show the performance of the unoptimized float32, or the quantized int8, version of the neural network.

Sort

Sorting options are available to find the parameters best suited to a given application or hardware target. For constrained devices, sort by RAM to show options with the smallest memory footprint, or sort by latency to find models with the lowest number of operations per inference. It's also possible to sort by label, finding the best model for identifying a specific class.

The selected sorting criteria will be shown in the top left corner of each result.

After training and validating your model, you can now deploy it to any device. This makes the model run without an internet connection, minimizes latency, and runs with minimal power consumption.

The Deployment page consists of a variety of deploy options to choose from depending on your target device. Regardless of whether you are using a fully supported development board or not, Edge Impulse provides deploy options through C++ library in which you can use to deploy your model on any targets (as long as the target has enough compute can handle the task).

The following are the 4 main categories of deploy options currently supported by Edge Impulse:

1. Deploy as a customizable library

2. Deploy as a pre-built firmware - for fully supported development boards

3. Run directly on your phone or computer

4. Use Edge Impulse for Linux for Linux targets

Deploying as a customizable library

This deploy option lets you turn your impulse into a fully optimized source code that can be further customized and integrated with your application. This option supports the following libraries:

Arduino Library

You can run your impulse locally as an Arduino library. This packages all of your signal processing blocks, configuration and learning blocks up into a single package.

To deploy as an Arduino library, select Arduino library on the Deployment page and click Build to create the library. Download the .ZIP file and import it as a sketch in your Arduino IDE then run your application.

For a full tutorial on how to run your impulse locally as an arduino library, have a look at Running your impulse locally - Arduino.

C++ Library

You can run your Impulse as a C++ library. This packages all of your signal processing blocks, configuration and learning blocks up into a single package that can be easily ported to your custom applications.

Visit Running your impulse locally for a deep dive on how to deploy your impulse as a C++ library.

Cube.MX CMSIS-PACK library

If you want to deploy your impulse to an STM32 MCU, you can use the Cube.MX CMSIS-PACK. This packages all your signal processing blocks, configuration and learning blocks up into a single package. You can include this package in any STM32 project with a single function call.

Have a look at Running your impulse locally - using CubeAI for a deep dive on how to deploy your impulse on STM32 based targets using the Cube.MX CMSIS-PACK.

WebAssembly Library

When you want to deploy your impulse to a web app you can use the WebAssembly library.This packages all your signal processing blocks, configuration and learning blocks up into a single package that can run without any compilation.

Have a look at Running your impulse locally - through WebAssembly (Browser) fora deep dive on how you can run your impulse to classify sensor data in your Node.js application.

Deploy as a pre-built firmware

For this option, you can use a ready-to-go binary for your development board that bundles signal processing blocks, configuration and learning blocks up into a single package. This option is currently only available for fully supported development boards as shown in the image below:

To deploy your model using ready to go binaries, select your target device and click "build". Flash the downloaded firmware to your device then run the following command:

 $ edge-impulse-run-impulse

The impulse runner shows the results of your impulse running on your development board. This only applies to ready-to-go binaries built from the studio.

Edge Impulse for Linux

If you are developing for Linux based devices, you can use Edge Impulse for Linux for deployment. It contains tools which let you collect data from any microphone or camera, can be used with the Node.js, Python, Go and C++ SDKs to collect new data from any sensor, and can run impulses with full hardware acceleration - with easy integration points to write your own applications.

For a deep dive on how to deploy your impulse to linux targets using Edge Impulse for linux, you can visit the Edge Impulse for Linux tutorial.

Deploy to your mobile phone/computer

You can run your impulse directly on your computer/mobile phone without the need of additional app. To run on your computer, you simply just need to select "computer" then click "Switch to classification mode". To run on your mobile phone, select 'Mobile Phone' then scan the QR code and click 'switch to classification mode".

Optimizations

Enabling EON Compiler

When building your impulse for deployment, Edge Impulse gives you the option of adding another layer of optimization to your impulse using the EON compiler. The EON Compiler lets you run neural networks in 25-55% less RAM, and up to 35% less flash, while retaining the same accuracy, compared to TensorFlow Lite for Microcontrollers.

To activate the EON Compiler, select you preferred deployment option then go to Enable EON™ Compiler then enable it and click 'Build' to build your impulse for deployment.

To have a peek of how your impulse would utilize compute resources of your target device, Edge Impulse also gives an estimate of latency, flash, RAM to be consumed by your target device even before deploying your impulse locally. This can really save you a lot of engineering time costs incurred by recurring iterations and experiments.

You can also select whether to run the unquantized float32 or the quantized int8 models as shown in the image below.

The above confusion matrix is only based on the test data to help you know how your model performs on unseen real world data. It can also help you know whether your model has learned to overfit on your training data which is a common occurrence.

Organizational datasets allow you to build a large collection of organized sensor data that is internal to your organization. This data can then be used to create new Edge Impulse projects, imported in Pandas or Matlab for internal exploration by your data scientists, or be processed and shared with partners. Data files within the datasets can be stored on-premise or in your own cloud infrastructure.

In this tutorial we'll set up a first dataset, explore the powerful query tool, and show how to create new Edge Impulse projects from raw data.

1. Configuring a storage bucket

Data is stored in storage buckets, which can either be hosted by Edge Impulse, or in your own infrastructure. If you choose to host the data yourself your infrastructure should be available through the S3 API, and you are responsible for setting up proper backups. To configure a new storage bucket, head to your organization, choose Data > Buckets, click Add new bucket, and fill in your access credentials. Make sure to name your storage bucket Internal datasets, as we'll need it to upload data later.

2. Uploading your first dataset

2.1 About datasets

With the storage bucket in place you can upload your first dataset. Datasets in Edge Impulse have three layers: 1) the dataset, a larger set of data items, grouped together. 2) data item, an item with metadata and files attached. 3) data file, the actual files. For example, if we're collecting data on physical activities from many subjects, we can have:

• Dataset: 'Activities Field Study September 1994'.

  • Data item: 'Forrest Gump Running', with metadata fields "name=Forrest Gump" and "activity=running".

    • Data file: 'running01.parquet', with raw sensor data.

    • Data file: 'running02.parquet', with raw sensor data.

From here you can query and group the data. For example, you can retrieve all data from the 'Activities Field Study September 1994' dataset that was tagged with the 'running' activity. Or, you can select all the files that are smaller than 1MB and were generated by 'Forrest Gump' over all datasets.

2.2 Importing the continuous gestures dataset

For this tutorial we'll use a dataset containing 9 minutes of accelerometer data for a gesture recognition system. Download the dataset and unzip it in a convenient location.

There are three ways of uploading data to your dataset. You can either:

1. Upload the files directly with the UI (we'll do this in this tutorial).

2. Upload data through the Edge Impulse API.

3. Or, upload data directly to the storage bucket (recommended for large datasets). In this case use Add data... > Add dataset from bucket and the data will be discovered automatically.

For this dataset we want to create four data items, one for every class ('idle', 'snake', 'updown', 'wave'). On the Data page, select Add data... > Add data item, set the name to 'Idle', the dataset to 'Gestures study', the metadata to { "gesture": "idle" }, and select all 'idle' files.

Do the same for the 'snake', 'updown' and 'wave' data, so you end up with four data items with 70 files in total.

3. Querying and downloading data

Organizational datasets contain a powerful query system which lets you explore and slice data. You control the query system through the 'Filter' text box, and you use a language which is very similar to SQL (documentation). For example, here are some queries that you can make:

• dataset = 'Gestures study' - returns all items and files from the study.

• bucket_name = 'Internal datasets' AND name IN ('Updown', 'Snake') - returns data whose name is either 'Updown' or 'Snake, and that is stored in the 'Internal datasets' bucket.

• metadata->gesture = 'updown' - return data that have a metadata field 'gesture' which contains 'updown'.

• created > DATE('2020-03-01') - returns all data that was created after March 1, 2020.

After you've created a filter, you can select one or more data items, and select Download selected to create a ZIP file with the data files. The file count reflects the number of files returned by the filter.

The previous queries all returned all files for a data item. But you can also query files through the same filter. In that case the data item will be returned, but only with the files selected. For example:

• file_name LIKE '%.0.cbor' - returns all files that end with .0.cbor.

If you have an interesting query that you'd like to share with your colleagues, you can just share the URL. The query is already added to it automatically.

3.1 All available fields

These are all the available fields in the query interface:

• dataset - Dataset.

• bucket_id - Bucket ID.

• bucket_name - Bucket name.

• bucket_path - Path of the data item within the bucket.

• id - Data item ID.

• name - Data item name.

• total_file_count - Number of files for the data item.

• total_file_size - Total size of all files for the data item.

• created - When the data item was created.

• metadata->key - Any item listed under 'metadata'.

• file_name - Name of a file.

• file_names - All filenames in the data item, that you can use in conjunction with CONTAINS. E.g. find all items with file X, but not file Y: file_names CONTAINS 'x' AND not file_names CONTAINS 'y'.

4. Importing data in an Edge Impulse project

If you have an interesting subset of data, and want to train a machine learning on this data, you can export the data into a new Edge Impulse project. This will make a copy of the data, that you can then manipulate and explore like any other project, or share with outside researchers without any risk of leaking the rest of your dataset. Data is also stripped of any metadata, like the name of the data item, or any metadata that you attached to the files.

Let's put this in practice. You need to select some data for the new project. Go to the Data page and set the filter to:

dataset = 'Gestures study'

Then, select all items and click Transform selected (70 files)

This redirects you to the 'Transformation job' page. Under 'Import data into', select 'Project'. Under 'Project' select '+ Create new project', and enter a name. Next, select the category. This determines whether this is 'training' or 'testing' data, or that the data should be split up between these two categories. For now, select 'Split'. Then, click Create project to import the data.

This pulls down the gesture data from the bucket, and then imports it into the project. You don't need to stay on the page, the job will continue running in the background.

If you now go back to your project you have a copy of the organizational dataset to your disposal, ready to build your next machine learning model. You can also add colleagues or outside collaborators to this specific project by going to Dashboard, and selecting the "Collaborators" widget. And if you want to do another experiment with the same data, you can easily create a new project with the same flow without any fear of changing any of the source data. 🚀

Any questions, or interested in the enterprise version of Edge Impulse? Contact us for more information.

Appendix: advanced features

Checklists

You can optionally show a check mark in the list of data items, and show a check list for data items. This can be used to quickly view which data items are complete (if you need to capture data from multiple sources) or whether items are in the right format.

Checklists are driven by the metadata for a data item. Set the ei_check metadata item to either 0 or 1 to show a check mark in the list. Set an ei_check_KEYNAME metadata item to 0 or 1 to show the item in the check list.

To query for items with or without a check mark, use a filter in the form of:

metadata->ei_check = 1

To make it easy to create these lists on the fly you can set these metadata items directly from a transformation block.

Within an organization you can work on one project with multiple people. These can be colleagues, outside researchers, or even members of the community. They will only get access to the specific data in the project, and not to any of the raw data in your organizational datasets.

To give someone access, go to your project's dashboard, and find the "Collaborators" widget. Click the '+' icon, and type the username or e-mail address of the other user. This user needs to have an Edge Impulse account already.

Upload portals are a secure way to let external parties upload data to your datasets. Through an upload portal they get an easy user interface to add data, but they have no access to the content of the dataset, nor can they delete any files. Data that is uploaded through the portal can be stored on-premise or in your own cloud infrastructure.

In this tutorial we'll set up an upload portal, show you how to add new data, and how to show this data in Edge Impulse for further processing.

1. Configuring a storage bucket

Data is stored in storage buckets, which can either be hosted by Edge Impulse, or in your own infrastructure. Follow the tutorial to set up your storage bucket.

2. Creating an upload portal

With your storage bucket configured you're ready to set up your first upload portal. In your organization go to Data > Upload portals and choose Create new upload portal. Here, select a name, a description, the storage bucket, and a path in the storage bucket.

Note: You'll need to enable CORS headers on the bucket. If these are not configured you'll get prompted with instructions. Talk to your user success engineer (when your data is hosted by Edge Impulse), or your system administrator to configure this.

After your portal is created a link is shown. This link contains an authentication token, and can be shared directly with the third party.

Click the link to open the portal. If you ever forget the link: no worries. Click the ⋮ next to your portal, and choose View portal.

3. Uploading data to the portal

To upload data you can now drag & drop files or folders to the drop zone on the right, or use Create new folder to first create a folder structure. There's no limit to the amount of files you can upload here, and all files are hashed, so if you upload a file that's already present the file will be skipped.

Note: Files with the same name but with a different hash are overwritten.

4. Adding the data to your dataset

To view the uploaded data in your dataset now go to your organization, and select Data. Then select Add data > Add dataset from bucket. Here, enter a name for the dataset, and select the same bucket path as you used for the portal. Then click Add data.

You now have the data in your Edge Impulse organization, ready to be applied to your next machine learning project.

5. Recap

Transformation blocks take raw data from your and convert the data into files that can be loaded in an Edge Impulse project. You can use transformation blocks to only include certain parts of individual data files, calculate long-running features like a running mean or derivatives, or efficiently generate features with different window lengths. Transformation blocks can be written in any language, and run on the Edge Impulse infrastructure.

In this tutorial we build a Python-based transformation block that loads Parquet files, splits the data into thirty second windows, and uploads the data to a new project.

Want more? We also have an end-to-end example transformation block that mixes noise into an audio dataset: .

1. Prerequisites

You'll need:

• The .

  • If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

• The file which you can use to test the transformation block. This contains some data from the dataset in Parquet format.

Transformation blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• installed on your machine.

1.1 - Parquet schema

This is the Parquet schema for the gestures.parquet file which we'll want to transform into data for a project:

message root {
  required binary sampleName (UTF8);
  required int64 timestamp (TIMESTAMP_MILLIS);
  required int64 added (TIMESTAMP_MILLIS);
  required boolean signatureValid;
  required binary device (UTF8);
  required binary label (UTF8);
  required float accX;
  required float accY;
  required float accZ;
}

2. Building your first transformation block

To build a transformation block open a command prompt or terminal window, create a new folder, and run:

$ edge-impulse-blocks init

This will prompt you to log in, and enter the details for your block. E.g.:

Edge Impulse Blocks v1.9.0
? What is your user name or e-mail address (edgeimpulse.com)? jan+demo@edgeimpulse.com
? What is your password? [hidden]
Attaching block to organization 'Demo org Inc.'
? Choose a type of block Transformation block
? Choose an option Create a new block
? Enter the name of your block Demo project transformation
? Enter the description of your block Reads a Parquet file and splits it up in labeled data
Creating block with config: {
  name: 'Demo project transformation',
  type: 'transform',
  description: 'Reads a Parquet file and splits it up in labeled data',
  organizationId: 34
}
Your new block 'Demo project transformation' has been created in '~/repos/tutorial-processing-block'.
When you have finished building your transformation block, run "edge-impulse-blocks push" to update the block in Edge Impulse.

Then, create the following files in this directory:

2.1 - Dockerfile

We're building a Python based transformation block. The Dockerfile describes our base image (Python 3.7.5), our dependencies (in requirements.txt) and which script to run (transform.py).

FROM python:3.7.5-stretch

WORKDIR /app

# Python dependencies
COPY requirements.txt ./
RUN pip3 --no-cache-dir install -r requirements.txt

COPY . ./

ENTRYPOINT [ "python3",  "transform.py" ]

Note: Do not use a WORKDIR under /home! The /home path will be mounted in by Edge Impulse, making your files inaccessible.

2.2 - requirements.txt

This file describes the dependencies for the block. We'll be using pandas and pyarrow to parse the Parquet file, and numpy to do some calculations.

numpy==1.16.4
pandas==0.23.4
pyarrow==0.16.0

2.3 - transform.py

This file includes the actual application. Transformation blocks are invoked with three parameters (as command line arguments):

• --in-file - A file from the organizational dataset. In this case the gestures.parquet file.

• --out-directory - Directory to write files to.

• --hmac-key - You can use this HMAC key to sign the output files.

Add the following content:

import pyarrow.parquet as pq
import numpy as np
import math, os, sys, argparse, json, hmac, hashlib, time
import pandas as pd

# these are the three arguments that we get in
parser = argparse.ArgumentParser(description='Organization transformation block')
parser.add_argument('--in-file', type=str, required=True)
parser.add_argument('--out-directory', type=str, required=True)
parser.add_argument('--hmac-key', type=str, required=True)

args, unknown = parser.parse_known_args()

# verify that the input file exists and create the output directory if needed
if not os.path.exists(args.in_file):
    print('--in-file argument', args.in_file, 'does not exist', flush=True)
    exit(1)

if not os.path.exists(args.out_directory):
    os.makedirs(args.out_directory)

# load and parse the input file
print('Loading parquet file', args.in_file, flush=True)
table = pq.read_table(args.in_file)
data = table.to_pandas()

# we'll split data based on the "label" Parquet column
# keep track of the last label and if it changes we write a file
window = []
last_label = data['label'][0]
interval_ms = (data['timestamp'][1] - data['timestamp'][0]).microseconds / 1000

# turn a window into a file that is in the Edge Impulse Data Acquisition format
def window_to_file(window):
    # window under 5 seconds: skip
    if (len(window) * interval_ms < 5000):
        return

    # take the label, timestamp and sensors
    label = window[0]['label']
    timestamp = int(pd.to_datetime(window[0]['timestamp']).value / 100000000)
    sensors = ['accX', 'accY', 'accZ']
    values = []
    for w in window:
        values.append([ w[x] for x in sensors ])

    # basic structure
    struct = {
        'protected': {
            'ver': 'v1',
            'alg': 'HS256',
            'iat': int(timestamp)
        },
        'signature': '0000000000000000000000000000000000000000000000000000000000000000',
        'payload': {
            'device_type': 'Importer',
            'interval_ms': interval_ms,
            'sensors': [ { 'name': x, 'units': 'm/s2' } for x in sensors ],
            'values': values
        }
    }

    # sign the structure
    encoded = json.dumps(struct)
    signature = hmac.new(bytes(args.hmac_key, 'utf-8'), msg = encoded.encode('utf-8'), digestmod = hashlib.sha256).hexdigest()
    struct['signature'] = signature

    # and write to the output directory
    file_name = os.path.join(args.out_directory, label + '.' + str(timestamp) + '.json')
    with open(file_name, 'w') as f:
        json.dump(struct, f)

# loop over all rows in the Parquet file
for index, row in data.iterrows():
    # file changes, or longer than 10 seconds? write file
    if ((last_label != row['label']) or (len(window) * interval_ms > 10000)):
        print('writing file', str(index) + '/' + str(10000), row['label'], flush=True)
        window_to_file(window)
        window = []

    last_label = row['label']
    window.append(row)

# write the last window
window_to_file(window)

2.4 - Testing the transformation block locally

On your local machine

To test the transformation block locally, if you have Python and all dependencies installed, just run:

$ python3 transform.py --in-file gestures.parquet --out-directory out/ --hmac-key 123

This generates a number of JSON files in the out/ directory. You can test the import into an Edge Impulse project via:

$ edge-impulse-uploader --clean out/*.json

Docker

You can also build the container locally via Docker, and test the block. The added benefit is that you don't need any dependencies installed on your local computer, and can thus test that you've included everything that's needed for the block. This requires Docker desktop to be installed.

First, build the container:

$ docker build -t test-org-transform-parquet .

Then, run the container (make sure gestures.parquet is in the same directory):

$ docker run --rm -v $PWD:/data test-org-transform-parquet --in-file /data/gestures.parquet --out-directory /data/out --hmac-key 0123

This generates a number of JSON files in the out/ directory. You can test the import into an Edge Impulse project via:

$ edge-impulse-uploader --clean out/*.json

3. Pushing the transformation block to Edge Impulse

With the block ready we can push it to your organization. Open a command prompt or terminal window, navigate to the folder you created earlier, and run:

$ edge-impulse-blocks push

This packages up your folder, sends it to Edge Impulse where it'll be built, and finally is added to your organization.

Edge Impulse Blocks v1.9.0
Archiving 'tutorial-processing-block'...
Archiving 'tutorial-processing-block' OK (2 KB) /var/folders/3r/fds0qzv914ng4t17nhh5xs5c0000gn/T/ei-transform-block-7812190951a6038c2f442ca02d428c59.tar.gz

Uploading block 'Demo project transformation' to organization 'Demo org Inc.'...
Uploading block 'Demo project transformation' to organization 'Demo org Inc.' OK

Building transformation block 'Demo project transformation'...
Job started
...
Building transformation block 'Demo project transformation' OK

Your block has been updated, go to https://studio.edgeimpulse.com/organization/34/data to run a new transformation

The transformation block is now available in Edge Impulse under Data transformation > Transformation blocks.

If you make any changes to the block, just re-run edge-impulse-blocks push and the block will be updated.

4. Uploading gestures.parquet to Edge Impulse

Next, upload the gestures.parquet file, by going to Data > Add data... > Add data item, setting name as 'Gestures', dataset to 'Transform tutorial', and selecting the Parquet file.

This makes the gestures.parquet file available from the Data page.

5. Starting the job

With the Parquet file in Edge Impulse and the transformation block configured you can now create a new job. Go to Data, and select the Parquet file by setting the filter to dataset = 'Transform tutorial'.

Click the checkbox next to the data item, and select Transform selected (1 file). On the 'Create transformation job' page, select 'Import data into Project'. Then under 'Project', select '+ Create new project' and enter a name. Under 'Transformation block' select the new transformation block.

Click Start transformation job to start the job. This pulls the data in, starts a transformation job and finally imports the data into the new project. If you have multiple files selected the transformations will also run in parallel.

6. Next steps

Appendix: Advanced features

Environmental variables

Transformation blocks get access to the following environmental variables, which let you authenticate with the Edge Impulse API. This way you don't have to inject these credentials into the block. The variables are:

• EI_API_KEY - an API key with 'member' privileges for the organization.

• EI_ORGANIZATION_ID - the organization ID that the block runs in.

• EI_API_ENDPOINT - the API endpoint (default: https://studio.edgeimpulse.com/v1).

is available for everyone but has to be self-hosted. If you want to host it on Edge Impulse infrastructures, you can do that within your organization interface.

In this tutorial, you'll learn how to use to push your custom DSP block to your organisation and how to make this processing block available in the Studio for all users in the organization.

The Custom Processing block we are using for this tutorial can be found here: . It is written in Python. Please note that one of the beauties with custom blocks is that you can write them in any language as we will host a Docker container and we are not tied to a specific runtime.

Prerequisites

You'll need:

• The . If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

• installed on your machine. Custom blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• A running with Docker.

Init and upload your custom DSP block

Inside your Custom DSP block folder, run the following command:

edge-impulse-blocks init --clean

The output will look like this:

? What is your user name or e-mail address (edgeimpulse.com)? 
? What is your password? [hidden]
Edge Impulse Blocks v1.14.3
Attaching block to organization 'Demo Team'

? Choose a type of block 
  Transformation block 
  Deployment block 
❯ DSP block 
  Transfer learning block 

? Enter the name of your block Edge: Detection

? Enter the description of your block: Edge Detection processing block using Canny filters in images

Creating block with config: {
  version: 1,
  config: {
    'edgeimpulse.com': {
      name: 'Edge Detection',
      type: 'dsp',
      description: 'Edge Detection processing block using Canny filters in images',
      organizationId: XXX,
      operatesOn: undefined,
      tlObjectDetectionLastLayer: undefined,
      tlOperatesOn: undefined
    }
  }
}
Your new block 'Edge Detection' has been created in '<PATH>'.
When you have finished building your dsp block, run 'edge-impulse-blocks push' to update the block in Edge Impulse.

Modify or update your custom code if needed and run the following command:

edge-impulse-blocks push            

The output will look similar to this:

Edge Impulse Blocks v1.14.3
? What port is your block listening on? 4446

Archiving 'edge-detection-processing-block'...
Archiving 'edge-detection-processing-block' OK (476 KB) /var/folders/7f/pfcmh61s3hg9c59qd0dkkw5w0000gn/T/ei-dsp-block-c729b4a3ff761b64629617c869e9d934.tar.gz

Uploading block 'Edge Detection' to organization 'Demo Team'...
Uploading block 'Edge Detection' to organization 'Demo Team' OK

Building dsp block 'Edge Detection'...
Job started
...
Building dsp block 'Edge Detection' OK

That's it, now your custom DSP block is hosted on your organization. To make sure it is up and running, in your organisation, go to Custom blocks->DSP and you will see the following screen:

Use your custom hosted DSP block in your projects

To use your DSP block, simply add it as a processing block in the Create impulse view:

Other resources

Transformation blocks take raw data from your ) and convert the data into files that can be loaded in an Edge Impulse project. You can use transformation blocks to only include certain parts of individual data files, calculate long-running features like a running mean or derivatives, or efficiently generate features with different window lengths. Transformation blocks can be written in any language, and run on the Edge Impulse infrastructure.

In this tutorial we build a Python-based transformation block that loads Parquet files, calculates features from the Parquet file, and then writes a new file back to your dataset.If you haven't done so, go through first.

1. Prerequisites

You'll need:

• The .

  • If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

• The file which you can use to test the transformation block. This contains some data from the dataset in Parquet format.

Transformation blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• installed on your machine.

1.1 - Parquet schema

This is the Parquet schema for the gestures.parquet file which we'll transform:

message root {
  required binary sampleName (UTF8);
  required int64 timestamp (TIMESTAMP_MILLIS);
  required int64 added (TIMESTAMP_MILLIS);
  required boolean signatureValid;
  required binary device (UTF8);
  required binary label (UTF8);
  required float accX;
  required float accY;
  required float accZ;
}

2. Building your first transformation block

To build a transformation block open a command prompt or terminal window, create a new folder, and run:

$ edge-impulse-blocks init

This will prompt you to log in, and enter the details for your block. E.g.:

Edge Impulse Blocks v1.9.0
? What is your user name or e-mail address (edgeimpulse.com)? jan+demo@edgeimpulse.com
? What is your password? [hidden]
Attaching block to organization 'Demo org Inc.'
? Choose a type of block Transformation block
? Choose an option Create a new block
? Enter the name of your block Demo dataset transformation
? Enter the description of your block Reads a Parquet file, extracts features, and writes the block back to the dataset
Creating block with config: {
  name: 'Demo dataset transformation',
  type: 'transform',
  description: 'Reads a Parquet file and splits it up in labeled data',
  organizationId: 34
}
Your new block 'Demo dataset transformation' has been created in '~/repos/tutorial-processing-block'.
When you have finished building your transformation block, run "edge-impulse-blocks push" to update the block in Edge Impulse.

Then, create the following files in this directory:

2.1 - Dockerfile

We're building a Python based transformation block. The Dockerfile describes our base image (Python 3.7.5), our dependencies (in requirements.txt) and which script to run (transform.py).

FROM python:3.7.5-stretch

WORKDIR /app

# Python dependencies
COPY requirements.txt ./
RUN pip3 --no-cache-dir install -r requirements.txt

COPY . ./

ENTRYPOINT [ "python3",  "transform.py" ]

Note: Do not use a WORKDIR under /home! The /home path will be mounted in by Edge Impulse, making your files inaccessible.

2.2 - requirements.txt

This file describes the dependencies for the block. We'll be using pandas and pyarrow to parse the Parquet file, and numpy to do some calculations.

numpy==1.16.4
pandas==0.23.4
pyarrow==0.16.0

2.3 - transform.py

This file includes the actual application. Transformation blocks are invoked with three parameters (as command line arguments):

• --in-file - A file from the organizational dataset. In this case the gestures.parquet file.

• --out-directory - Directory to write files to.

• --hmac-key - You can use this HMAC key to sign the output files. This is not used in this tutorial.

• --metadata - Additional key/value pairs defined for the incoming item(s). This is not used in this tutorial.

Add the following content. This takes in the Parquet file, groups data by their label, and then calculates the RMS over the X, Y and Z axes of the accelerometer.

import pyarrow.parquet as pq
import numpy as np
import math, os, sys, argparse, json, hmac, hashlib, time
import pandas as pd

# these are the three arguments that we get in
parser = argparse.ArgumentParser(description='Organization transformation block')
parser.add_argument('--in-file', type=str, required=True)
parser.add_argument('--out-directory', type=str, required=True)

args, unknown = parser.parse_known_args()

# verify that the input file exists and create the output directory if needed
if not os.path.exists(args.in_file):
    print('--in-file argument', args.in_file, 'does not exist', flush=True)
    exit(1)

if not os.path.exists(args.out_directory):
    os.makedirs(args.out_directory)

# load and parse the input file
print('Loading parquet file', args.in_file, flush=True)
table = pq.read_table(args.in_file)
data = table.to_pandas()

features = []

# we group by label and then extract some metrics
for label in data.label.unique():
    data_per_label = data[data.label == label]

    # calculate the RMS per axis
    features.append({
        'label': label,
        'rmsX': np.sqrt(np.mean(data_per_label.accX**2)),
        'rmsY': np.sqrt(np.mean(data_per_label.accY**2)),
        'rmsZ': np.sqrt(np.mean(data_per_label.accZ**2))
    })

# and store as new file in the output directory
out_file = os.path.join(args.out_directory, os.path.splitext(os.path.basename(args.in_file))[0] + '_features.parquet')
pd.DataFrame(features).to_parquet(out_file)

print('Written features file', out_file, flush=True)

2.4 - Building and testing the container

On your local machine

To test the transformation block locally, if you have Python and all dependencies installed, just run:

$ python3 transform.py --in-file gestures.parquet --out-directory out/

Docker

You can also build the container locally via Docker, and test the block. The added benefit is that you don't need any dependencies installed on your local computer, and can thus test that you've included everything that's needed for the block. This requires Docker desktop to be installed.

To build the container and test the block, open a command prompt or terminal window and navigate to the source directory. First, build the container:

$ docker build -t test-org-transform-parquet-dataset .

Then, run the container (make sure gestures.parquet is in the same directory):

$ docker run --rm -v $PWD:/data test-org-transform-parquet-dataset --in-file /data/gestures.parquet --out-directory /data/out

Seeing the output

This process has generated a new Parquet file in the out/ directory containing the RMS of the X, Y and Z axes. If you inspect the content of the file (e.g. using parquet-tools) you'll see the output:

$ parquet-tools head -n5 out/gestures_features.parquet 
label = wave
rmsX = 11.424144744873047
rmsY = 4.73303747177124
rmsZ = 2.944265842437744

label = updown
rmsX = 3.899503231048584
rmsY = 3.9587674140930176
rmsZ = 10.34404468536377

label = circle
rmsX = 6.263721942901611
rmsY = 7.0987162590026855
rmsZ = 6.159618854522705

label = idle
rmsX = 3.714001178741455
rmsY = 3.4940428733825684
rmsZ = 8.6710205078125

label = snake
rmsX = 1.282995581626892
rmsY = 1.8830623626708984
rmsZ = 9.597149848937988

Success!

3. Pushing the transformation block to Edge Impulse

With the block ready we can push it to your organization. Open a command prompt or terminal window, navigate to the folder you created earlier, and run:

$ edge-impulse-blocks push

This packages up your folder, sends it to Edge Impulse where it'll be built, and finally is added to your organization.

Edge Impulse Blocks v1.9.0
Archiving 'tutorial-processing-block'...
Archiving 'tutorial-processing-block' OK (2 KB) /var/folders/3r/fds0qzv914ng4t17nhh5xs5c0000gn/T/ei-transform-block-7812190951a6038c2f442ca02d428c59.tar.gz

Uploading block 'Demo dataset transformation' to organization 'Demo org Inc.'...
Uploading block 'Demo dataset transformation' to organization 'Demo org Inc.' OK

Building transformation block 'Demo dataset transformation'...
Job started
...
Building transformation block 'Demo dataset transformation' OK

Your block has been updated, go to https://studio.edgeimpulse.com/organization/34/data to run a new transformation

The transformation block is now available in Edge Impulse under Data transformation > Transformation blocks.

If you make any changes to the block, just re-run edge-impulse-blocks push and the block will be updated.

4. Uploading gestures.parquet to Edge Impulse

Next, upload the gestures.parquet file, by going to Data > Add data... > Add data item, setting name as 'Gestures', dataset to 'Transform tutorial', and selecting the Parquet file.

This makes the gestures.parquet file available from the Data page.

5. Starting the transformation

With the Parquet file in Edge Impulse and the transformation block configured you can now create a new job. Go to Data, and select the Parquet file by setting the filter to dataset = 'Transform tutorial'.

Click the checkbox next to the data item, and select Transform selected (1 file). On the 'Create transformation job' page select 'Import data into Dataset'. Under 'output dataset', select 'Same dataset as source', and under 'Transformation block' select the new transformation block.

Click Start transformation job to start the job. This pulls the data in, starts a transformation job and finally uploads the data back to your dataset. If you have multiple files selected the transformations will also run in parallel.

You can now find the transformed file back in your dataset:

6. Next steps

Appendix: Advanced features

Updating metadata from a transformation block

You can update the metadata of blocks directly from a transformation block by creating a ei-metadata.json file in the output directory. The metadata is then applied to the new data item automatically when the transform job finishes. The ei-metadata.json file has the following structure:

{
    "version": 1,
    "action": "add",
    "metadata": {
        "some-key": "some-value"
    }
}

Some notes:

• If action is set to add the metadata keys are added to the data item. If action is set to replace all existing metadata keys are removed.

Environmental variables

Transformation blocks get access to the following environmental variables, which let you authenticate with the Edge Impulse API. This way you don't have to inject these credentials into the block. The variables are:

• EI_API_KEY - an API key with 'member' privileges for the organization.

• EI_ORGANIZATION_ID - the organization ID that the block runs in.

• EI_API_ENDPOINT - the API endpoint (default: https://studio.edgeimpulse.com/v1).

Transformation blocks take raw data from your and convert the data into files that can be loaded in an Edge Impulse project. You can use transformation blocks to only include certain parts of individual data files, calculate long-running features like a running mean or derivatives, or efficiently generate features with different window lengths. Transformation blocks can be written in any language, and run on the Edge Impulse infrastructure.

Transformation blocks can output data to either:

1. A project - here the data in your dataset is placed in an Edge Impulse project, and you'll have all the normal features from the studio available to build your machine learning models. This is great if you already have an idea what you want with the data, and are looking for a reproducible pipeline.

2. Back in the dataset - here data is placed back in the dataset. This is great for extracting long running features, batch jobs, or combining data from multiple sources - even when you don't want to place the data in a project yet.

For both of these you can find tutorials here:

One of the most powerful features in Edge Impulse are the built-in deployment targets (under Deployment in the Studio), which let you create ready-to-go binaries for development boards, or custom libraries for a wide variety of targets that incorporate your trained impulse. You can also create custom deployment blocks for your organization. This lets developers quickly iterate on products without getting your embedded engineers involved, lets your customers build personalized firmware using their own data, or lets you create custom libraries.

In this tutorial you'll learn how to use custom deployment blocks to create a new deployment target, and how to make this target available in the Studio for all users in the organization.

Prerequisites

You'll need:

• The .

  • If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

Deployment blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• installed on your machine.

Then, create a new folder on your computer named custom-deploy-block.

1. Getting basic deployment info

When a user deploys with a custom deployment block two things happen:

1. A package is created that contains information about the deployment (like the sensors used, frequency of the data, etc.), any trained neural network in .tflite and SavedModel formats, the Edge Impulse SDK, and all DSP and ML blocks as C++ code.

2. This package is then consumed by the custom deployment block, which can incorporate it with a base firmware, or repackage it into a new library.

To obtain this package go to your project's Dashboard, look for Administrative zone, enable Custom deploys, and click Save.

If you now go to the Deployment page, a new option appears under 'Create library':

Once you click Build you'll receive a ZIP file containing five items:

• trained.tflite - if you have a neural network in the project this contains neural network in .tflite format. This network is already fully quantized if you choose the int8 optimization, otherwise this is the float32 model.

• trained.savedmodel.zip - if you have a neural network in the project this contains the full TensorFlow SavedModel. Note that we might update the TensorFlow version used to train these networks at any time, so rely on the compiled model or the TFLite file where possible.

• model-parameters - impulse and block configuration in C++ format. Can be used by the SDK to quickly run your impulse.

• tflite-model - neural network as source code in a way that can be used by the SDK to quickly run your impulse.

Store the unzipped file under custom-deploy-block/input.

2. Building a new binary

With the basic information in place we can create a new deployment block. Here we'll build a standalone application that runs our impulse on Linux, very useful when running your impulse on a gateway or desktop computer. First, open a command prompt or terminal window, navigate to the custom-deploy-block folder (that you created under 1.), and run:

$ edge-impulse-blocks init

This will prompt you to log in, and enter the details for your block.

2. Unzip under custom-deploy-block/app.

To build this application we need to combine the application with the edge-impulse-sdk, model-parameters and tflite-model folder, and invoke the (already included) Makefile.

2.1 Creating a build script

To build the application we use Docker, a virtualization technique which lets developers package up an application with all dependencies in a single package. In this container we'll place the build tools required for this application, and scripts to combine the trained impulse with the base application.

First, let's create a small build script. As a parameter you'll receive --metadata which points to the deployment information. In here you'll also get information on the input and output folders where you need to read from and write to.

Create a new file called custom-deploy-block/build.py and add:

build.py

import argparse, json, os, shutil, zipfile, threading

# parse arguments (--metadata FILE is passed in)
parser = argparse.ArgumentParser(description='Custom deploy block demo')
parser.add_argument('--metadata', type=str)
args = parser.parse_args()

# load the metadata.json file
with open(args.metadata) as f:
    metadata = json.load(f)

# now we have two folders 'metadata.folders.input' - this is where all the SDKs etc are,
# and 'metadata.folders.output' - this is where we need to write our output
input_dir = metadata['folders']['input']
app_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'app')
output_dir = metadata['folders']['output']

print('Copying files to build directory...')

is_copying = True
def print_copy_progress():
    if (is_copying):
        threading.Timer(2.0, print_copy_progress).start()
        print("Still copying...")
print_copy_progress()

# create a build directory, the input / output folders are on network storage so might be very slow
build_dir = '/tmp/build'
if os.path.exists(build_dir):
    shutil.rmtree(build_dir)
os.makedirs(build_dir)

# copy in the data from both 'input' and 'app' folders
os.system('cp -r ' + input_dir + '/* ' + build_dir)
os.system('cp -r ' + app_dir + '/* ' + build_dir)

is_copying = False

print('Copying files to build directory OK')
print('')

print('Compiling application...')

is_compiling = True
def print_compile_progress():
    if (is_compiling):
        threading.Timer(2.0, print_compile_progress).start()
        print("Still compiling...")
print_compile_progress()

# then invoke Make
os.chdir(build_dir)
os.system('make -f Makefile.tflite')

is_compiling = False

print('Compiling application OK')

# ZIP the build folder up, and copy to output dir
if not os.path.exists(output_dir):
    os.makedirs(output_dir)
shutil.make_archive(os.path.join(output_dir, 'deploy'), 'zip', os.path.join(build_dir, 'build'))

Next, we need to create a Dockerfile, which contains all dependencies for the build. These include GNU Make, a compiler, and both the build script and the base application.

Create a new file called custom-deploy-block/Dockerfile and add:

Dockerfile

FROM ubuntu:18.04

WORKDIR /ei

# Install base dependencies
RUN apt update && apt install -y build-essential software-properties-common wget

# Install LLVM 9
RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 9
RUN rm /usr/bin/gcc && rm /usr/bin/g++ && ln -s $(which clang-9) /usr/bin/gcc && ln -s $(which clang++-9) /usr/bin/g++

# Install Python 3.7
RUN apt install -y python3.7

# Copy the base application in
COPY app ./app

# Copy any scripts in that we have
COPY *.py ./

# This is the script our application should run (-u to disable buffering)
ENTRYPOINT [ "python3", "-u", "build.py" ]

2.2 Testing the build script with Docker

To test the build script we first build the container, then invoke it with the files from the input directory. Open a command prompt or terminal, navigate to the custom-deploy-block folder and:

1. Build the container:

$ docker build -t cdb-demo .

1. Invoke the build script - this mounts the current directory in the container under /home, and then passes the downloaded metadata script to the container:

$ docker run --rm -it -v $PWD:/home cdb-demo --metadata /home/input/deployment-metadata.json

$ ./output/edge-impulse-standalone "RAW FEATURES HERE"

Or if you run Windows or macOS, you can use Docker to run this application:

$ docker run --rm -v $PWD/output:/home ubuntu:18.04 /home/edge-impulse-standalone "RAW FEATURES HERE"

3. Uploading the deployment block to Edge Impulse

With the deployment block ready you can make it available in Edge Impulse. Open a command prompt or terminal window, navigate to the folder you created earlier, and run:

$ edge-impulse-blocks push

This packages up your folder, sends it to Edge Impulse where it'll be built, and finally is added to your organization. The transformation block is now available in Edge Impulse under Deployment blocks. You can go here to set the logo, update the description, and set extra command line parameters.

Privileged mode

4. Using the deployment block

The deployment block is automatically available for all organizational projects. Go to the Deployment page on a project, and you'll find a new section 'Custom targets'. Select your new deployment target and click Build.

And now you'll have a freshly built binary from your own deployment block!

5. Conclusion

Custom deployment blocks are a powerful tool for your organization. They let you build binaries for unreleased products, let you package up impulse as custom libraries, or can let your customers deploy to private targets (if you add an external collaborator to a project they'll have access to the blocks as well). Because the deployment blocks are integrated with your project, and hosted by Edge Impulse this lets everyone, from FAE to R&D developer, now iterate on on-device models without getting your embedded engineers involved.

This is the specification for the deployment-metadata.json file from Building deployment blocks.

export interface DeploymentMetadataV1 {
    version: 1;
    // Global deployment counter
    deployCounter: number;
    // The output classes (for classification)
    classes: string[];
    // The number of samples to be taken per inference (e.g. 100Hz data, 3 axis, 2 seconds => 200)
    samplesPerInference: number;
    // Number of axes ((e.g. 100Hz data, 3 axis, 2 seconds => 3)
    axesCount: number;
    // Frequency of the data
    frequency: number;
    // TFLite models (already converted and quantized)
    tfliteModels: {
        // Information about the model type, e.g. quantization parameters
        details: KerasModelIODetails;
        // Name of the input tensor
        inputTensor: string;
        // Name of the output tensor
        outputTensor: string;
        // Path of the model on disk
        modelPath: string;
        // Calculated arena size when running TFLite in interpreter mode
        arenaSize: number;
        // Number of values to be passed into the model
        inputFrameSize: number;
    }[];
    // Project information
    project: {
        name: string;
        // API key, only set for deploy blocks with privileged flag and development keys set
        apiKey: string | undefined;
    };
    // Impulse information
    impulse: DeploymentMetadataImpulse;
    // Sensor guess based on the input
    sensor: 'camera' | 'microphone' | 'accelerometer' | undefined;
    // Folder locations
    folders: {
        // Input files are here, the input folder contains 'edge-impulse-sdk', 'model-parameters', 'tflite-model'
        input: string;
        // Write your output file here
        output: string;
    };
}

export type ResizeEnum = 'squash' | 'fit-short' | 'fit-long' | 'crop';
export type CropAnchorEnum = 'top-left' | 'top-center' | 'top-right' |
                             'middle-left' | 'middle-center' | 'middle-right' |
                             'bottom-left' | 'bottom-center' | 'bottom-right';

export interface CreateImpulseStateInput {
    id: number;
    type: 'time-series' | 'image';
    name: string;
    title: string;
    windowSizeMs?: number;
    windowIncreaseMs?: number;
    imageWidth?: number;
    imageHeight?: number;
    resizeMode?: ResizeEnum;
    cropAnchor?: CropAnchorEnum;
}

export interface CreateImpulseStateDsp {
    id: number;
    type: string | 'custom';
    name: string;
    axes: string[];
    title: string;
    customUrl?: string;
}

export interface CreateImpulseStateLearning {
    id: number;
    type: string;
    name: string;
    dsp: number[];
    title: string;
}

export interface CreateImpulseState {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: CreateImpulseStateDsp[];
    learnBlocks: CreateImpulseStateLearning[];
}

export interface DSPConfig {
    options: { [k: string ]: string | number | boolean };
}

export type DSPFeatureMetadataOutput = {
    type: 'image',
    shape: { width: number, height: number, channels: number }
} | {
    type: 'spectrogram',
    shape: { width: number, height: number }
} | {
    type: 'flat',
    shape: { width: number }
};

export interface DSPFeatureMetadata {
    created: Date;
    dspConfig: DSPConfig;
    labels: string[];   // the training labels
    featureLabels: string[];
    valuesPerAxis: number;
    windowCount: number;
    windowSizeMs: number;
    windowIncreaseMs: number;
    frequency: number;
    includedSamples: { id: number, windowCount: number }[];
    outputConfig: DSPFeatureMetadataOutput;
}

/**
 * Information necessary to quantize or dequantize the contents of a tensor
 */
export type KerasModelTensorDetails = {
    dataType: 'float32'
} | {
    dataType: 'int8';
    // Scale and zero point are used only for quantized tensors
    quantizationScale?: number;
    quantizationZeroPoint?: number;
};

export type KerasModelTypeEnum = 'int8' | 'float32' | 'requiresRetrain';

/**
 * Information required to process a model's input and output data
 */
export interface KerasModelIODetails {
    modelType: KerasModelTypeEnum;
    inputs: KerasModelTensorDetails[];
    outputs: KerasModelTensorDetails[];
}

export interface DeploymentMetadataImpulse {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: (CreateImpulseStateDsp & { metadata: DSPFeatureMetadata | undefined })[];
    learnBlocks: CreateImpulseStateLearning[];
}

There is a list of development boards that are fully supported by Edge Impulse. These boards come with a special firmware which enables data collection from all their sensors, allows you to build new ready-to-go binaries that include your trained impulse, and come with examples on integrating your impulse with your custom firmware. These boards are the perfect way to start building Machine Learning solutions on real embedded hardware.

Officially supported MCU targets

• Arduino Nano 33 BLE Sense

• Arduino Nicla Sense ME

• Arduino Nicla Vision

• Arduino Portenta H7 + Vision Shield

• Espressif ESP32

• Himax WE-I Plus

• Nordic Semi nRF52840 DK

• Nordic Semi nRF5340 DK

• Nordic Semi nRF9160 DK

• Nordic Semi Thingy:91

• Open MV Cam H7 Plus

• Silicon Labs xG24 Dev Kit

• Silicon Labs Thunderboard Sense 2

• Sony's Spresense

• ST B-L475E-IOT01A

• Syntiant Tiny ML Board

• TI CC1352P Launchpad

• Raspberry Pi RP2040

Officially supported CPU/GPU targets

• Intel Based Macs

• Linux x86_64

• NVIDIA Jetson Nano

• Raspberry Pi 4

Community boards

• Seeed Wio Terminal

• Arducam Pico4ML TinyML Dev Kit

• Blues Wireless Swan

Different development board? No problem, you can always collect data using the Data forwarder or the Edge Impulse for Linux SDK, and deploy your model back to the device with the Running your impulse locally tutorials. Also, if you feel like porting your board, use this Porting guide.

Just want to experience Edge Impulse? You can also use your Mobile phone!

Enterprise customers can add fully custom learning models in Edge Impulse. These models can be represented in any deep learning framework as long as it can output TFLite files.

Custom learning blocks for organizations go beyond project's expert mode, which lets you to design your own neural network architectures on a project by project basis, but has some caveats:

• You can't load pretrained weights from expert mode, and

• Your expert mode model needs to be representable in Keras / TensorFlow.

This tutorial describes how to build these models. Alternatively, we've put together two example projects, which bring these models into Edge Impulse:

• YOLOv5 - brings a YOLOv5 transfer learning model (trained with PyTorch) into Edge Impulse

• Keras - shows how to bring custom Keras blocks into Edge Impulse.

• PyTorch - shows how to bring custom PyTorch blocks into Edge Impulse.

Prerequisites

• The Edge Impulse CLI

  • If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

• Docker desktop - Custom learning models use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package.

Data formats

To bring custom learning models into Edge Impulse you'll need to encapsulate your training pipeline into a container. This container takes in the training data, trains the model and then spits out TFLite files.

To see the data that will be passed into the container:

1. Create a project in Edge Impulse and add your data.

2. Under Create impulse add the DSP block that you'll use (e.g. 'Image' for images, or 'Spectrogram' for audio) and a random neural network block.

3. Generate features for the DSP block.

4. Now go to Dashboard, and under 'Download block data' grab the two items marked 'Training data' / 'Training labels'.

This data is in the following formats:

• Training data: a numpy matrix with one sample per row containing the output of the DSP block (use np.load('X_train_features.npy') to see).

• Training labels

  • If you're using object detection: a JSON file (despite the '.npy' extension) containing structured information about the bounding boxes. Every item in the samples array maps to one row in the data matrix. The label is the index of the class, and is 1-based.

  • If you're not using object detection: a numpy matrix with one sample per row, and the first column of every row is the index of the class. The last three columns are the sampleId and the start / end time of the sample (when going through time series). During training you can typically discard this.

This data is passed into the container as files, located here:

• Data: /home/X_train_features.npy

• Labels: /home/y_train.npy

After training you need to output a TFLite file. You need to write these here:

• Float32 (unquantized) model: /home/model.tflite

• Int8 quantized model with int8 inputs: /home/model_quantized_int8_io.tflite

Wrapping your training scripts in a Docker container

Docker containers are a virtualization technique which lets developers package up an application with all dependencies in a single package. To train your custom model you'll need to wrap all the required packages, your scripts, and (if you use transfer learning) your pretrained weights into this container. When running in Edge Impulse the container does not have network access, so make sure you don't download dependencies while running (fine when building the container).

A typical Dockerfile might look like:

# syntax = docker/dockerfile:experimental
FROM ubuntu:20.04
WORKDIR /app

ARG DEBIAN_FRONTEND=noninteractive

# Install base packages (like Python and pip)
RUN apt update && apt install -y curl zip git lsb-release software-properties-common apt-transport-https vim wget python3 python3-pip
RUN python3 -m pip install --upgrade pip==20.3.4

# Copy Python requirements in and install them
COPY requirements.txt ./
RUN pip3 install -r requirements.txt

# Copy the rest of your training scripts in
COPY . ./

# And tell us where to run the pipeline
ENTRYPOINT ["python3", "-u", "train.py"]

It's important to create an ENTRYPOINT at the end of the Dockerfile to specify which file to run.

Arguments during training