Custom blocks are cloud jobs that can be hosted and used on Edge Impulse. They serve a dedicated task, are extremely flexible, let you customize your experience and fasten your time-to-market.

• Creating a transformation block - to fetch, sort, validate, combine and transform existing data into robust datasets that can be imported into your projects.

• Building and hosting custom DSP blocks - to create and host your custom signal processing techniques and use them directly in your projects.

• Create a custom learning block - to use your custom models and load pre-trained weights with PyTorch, Keras or scikit-learn.

• Building deployment blocks - to create custom deployment targets for your products.

Transformation blocks are very flexible and can be used for most advanced use cases.

They can either take raw data from your and convert the data into files that can be loaded in an Edge Impulse project/another organizational dataset. But you can also use the transformation blocks as cloud jobs to perform specific actions using standalone mode.

Transformation blocks are available in your and in your so you can automate your processes.

You can use transformation blocks to fetch external datasets, augment/create variants of your data samples, generate synthetic datasets, extract metadata from config files, create helper graphs, align and interpolate measurements across sensors, or remove duplicate entries. The possibilities are endless.

Transformation blocks can be written in any language, and run on Edge Impulse infrastructure.

Run transformation blocks

You can run your transformation blocks as transformation jobs. They can be triggered:

from your organization:

• From this view, Custom blocks->Transformation

from your projects:

Public blocks

By default, we provide several pre-built transformation blocks that you can use directly in your organization or your organization's projects.

We will add more over time when we see a recurring need or interest. The current ones are the following:

Understanding the transformation blocks

A transformation block consists of a Docker image that contains one or several scripts. The Docker image is encapsulated in the transformation block with additional parameters.

Here is a minimal configuration for the transformation blocks:

In this documentation page, we will explain how to setup a transformation block and will explain the different options.

Import existing transformation blocks

You can directly create your transformation block within Edge Impulse Studio from a public Docker image or import existing transformation blocks:

Setting up transformation blocks

Tip: If you want to access your bucket, make sure to press <space> to select the bucket attached to your organization.

The step above will create the following .ei-block-config in your project directory:

To push your transformation block, simply run edge-impulse-blocks push.

Dockerfile

At Edge Impulse, we mostly use Python, Javascript/Typescript and Bash scripts, but you can write your transformation blocks in any language.

Dockerfile example to trigger a Python script and install the required dependencies:

The Dockerfile above describes a base image (Python 3.7.5), the Python dependencies (in requirements.txt) and which script to run (transform.py).

Note: Do not use a WORKDIR under /home! The /home path will be mounted in by Edge Impulse, making your files inaccessible.

Operation modes

We provide three modes to access your data:

• In the Standalone mode, no data is passed to the container, but you can still access data by mounting your bucket onto the container.

• At the Data item level, we pass the --in-directory and --out-directory arguments. The transformation jobs will run on each directory present in your selected path. These jobs can run in parallel.

• At the file level, we pass the --in-file and --out-directory arguments. The transformation jobs will run on each file present in your selected path. These jobs can run in parallel.

Standalone

The stand-alone method is the most flexible option (it can work on both generic and clinical datasets). You can consider this transformation block as a cloud job that you can use for anything in your machine learning pipelines.

Please note that this mode does not support running jobs in parallel, as it is unknown in advance how many files or how many directories are present in your dataset.

To access your data, you must mount your bucket/upload portal into the container, you can do this both when setting up your transformation block using Edge Impulse CLI, or directly in the studio when creating/editing a transformation block.

Examples

Data item (--in-directory)

When selecting the Data item operation mode, two parameters will be passed to the container:

• --in-directory

• --out-directory

The transformation jobs will run on each "Data item" (directory) present in your selected path or dataset.

File (--in-file)

When selecting the File operation mode, two parameters will be passed to the container:

• --in-file

• --out-directory

The transformation jobs will run on each file present in selected path.

Compute requests & limits

When editing your block on Edge Impulse Studio, you can set the number of desired CPUs and the memory needed for your container to run properly. Likely, you can set the limits of the same parameters.

Metadata (Data item and file operation modes)

You can update the metadata of blocks directly from a transformation block by creating a ei-metadata.json file in the output directory. The metadata is then applied to the new data item automatically when the transform job finishes. The ei-metadata.json file has the following structure:

Some notes:

• If action is set to add the metadata keys are added to the data item. If action is set to replace all existing metadata keys are removed.

Mounting points

When using the CLI to setup your block, by default we mount your bucket with the following mounting point:

You can change this value if you want your transformation block to behave differently.

Custom parameters

Environmental variables

Transformation blocks get access to the following environmental variables, which let you authenticate with the Edge Impulse API. This way you don't have to inject these credentials into the block. The variables are:

• EI_API_KEY - an API key with 'member' privileges for the organization.

• EI_ORGANIZATION_ID - the organization ID that the block runs in.

• EI_API_ENDPOINT - the API endpoint (default: https://studio.edgeimpulse.com/v1).

Examples & resources

Standalone

File (--in-file)

Data Item (--in-directory)

Recap

Now that you have a better idea of what are transformation blocks, here is a graphical recap of how it works:

Troubleshooting

The job run indefinitely

If you notice that your jobs run indefinitely, it is probably because of an error or the script has not been properly terminated. Make sure to exit your script with code 0 (return 0, exit(0) or sys.exit(0)) for success or with any other error code for failure.

Cannot access files in bucket

If you cannot access your files in your bucket, make sure that the mount point is properly configured.

When using the CLI, it is a common mistake to forget pressing <space> key to select the bucket attached to your organization.

Job failed without logs (only Job failed)

It probably means that we had an issue when triggering the container. In many cases it is related with the issue above, the mount point not being properly configured.

I cannot access the logs

We are still investigating why all the logs are not displayed properly. If you are using Python, you can also flush stdout after you print it using something like print("hello", flush=True).

Can I host my Docker image on Docker Hub?

It will print "hello +name" on the transformation job logs.

This is the specification for the deployment-metadata.json file from Building deployment blocks.

export interface DeploymentMetadataV1 {
    version: 1;
    // Global deployment counter
    deployCounter: number;
    // The output classes (for classification)
    classes: string[];
    // The number of samples to be taken per inference (e.g. 100Hz data, 3 axis, 2 seconds => 200)
    samplesPerInference: number;
    // Number of axes ((e.g. 100Hz data, 3 axis, 2 seconds => 3)
    axesCount: number;
    // Frequency of the data
    frequency: number;
    // TFLite models (already converted and quantized)
    tfliteModels: {
        // Information about the model type, e.g. quantization parameters
        details: KerasModelIODetails;
        // Name of the input tensor
        inputTensor: string;
        // Name of the output tensor
        outputTensor: string;
        // Path of the model on disk
        modelPath: string;
        // Calculated arena size when running TFLite in interpreter mode
        arenaSize: number;
        // Number of values to be passed into the model
        inputFrameSize: number;
    }[];
    // Project information
    project: {
        name: string;
        // API key, only set for deploy blocks with privileged flag and development keys set
        apiKey: string | undefined;
    };
    // Impulse information
    impulse: DeploymentMetadataImpulse;
    // Sensor guess based on the input
    sensor: 'camera' | 'microphone' | 'accelerometer' | undefined;
    // Folder locations
    folders: {
        // Input files are here, the input folder contains 'edge-impulse-sdk', 'model-parameters', 'tflite-model'
        input: string;
        // Write your output file here
        output: string;
    };
}

export type ResizeEnum = 'squash' | 'fit-short' | 'fit-long' | 'crop';
export type CropAnchorEnum = 'top-left' | 'top-center' | 'top-right' |
                             'middle-left' | 'middle-center' | 'middle-right' |
                             'bottom-left' | 'bottom-center' | 'bottom-right';

export interface CreateImpulseStateInput {
    id: number;
    type: 'time-series' | 'image';
    name: string;
    title: string;
    windowSizeMs?: number;
    windowIncreaseMs?: number;
    imageWidth?: number;
    imageHeight?: number;
    resizeMode?: ResizeEnum;
    cropAnchor?: CropAnchorEnum;
}

export interface CreateImpulseStateDsp {
    id: number;
    type: string | 'custom';
    name: string;
    axes: string[];
    title: string;
    customUrl?: string;
}

export interface CreateImpulseStateLearning {
    id: number;
    type: string;
    name: string;
    dsp: number[];
    title: string;
}

export interface CreateImpulseState {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: CreateImpulseStateDsp[];
    learnBlocks: CreateImpulseStateLearning[];
}

export interface DSPConfig {
    options: { [k: string ]: string | number | boolean };
}

export type DSPFeatureMetadataOutput = {
    type: 'image',
    shape: { width: number, height: number, channels: number }
} | {
    type: 'spectrogram',
    shape: { width: number, height: number }
} | {
    type: 'flat',
    shape: { width: number }
};

export interface DSPFeatureMetadata {
    created: Date;
    dspConfig: DSPConfig;
    labels: string[];   // the training labels
    featureLabels: string[];
    valuesPerAxis: number;
    windowCount: number;
    windowSizeMs: number;
    windowIncreaseMs: number;
    frequency: number;
    includedSamples: { id: number, windowCount: number }[];
    outputConfig: DSPFeatureMetadataOutput;
}

/**
 * Information necessary to quantize or dequantize the contents of a tensor
 */
export type KerasModelTensorDetails = {
    dataType: 'float32'
} | {
    dataType: 'int8';
    // Scale and zero point are used only for quantized tensors
    quantizationScale?: number;
    quantizationZeroPoint?: number;
};

export type KerasModelTypeEnum = 'int8' | 'float32' | 'requiresRetrain';

/**
 * Information required to process a model's input and output data
 */
export interface KerasModelIODetails {
    modelType: KerasModelTypeEnum;
    inputs: KerasModelTensorDetails[];
    outputs: KerasModelTensorDetails[];
}

export interface DeploymentMetadataImpulse {
    inputBlocks: CreateImpulseStateInput[];
    dspBlocks: (CreateImpulseStateDsp & { metadata: DSPFeatureMetadata | undefined })[];
    learnBlocks: CreateImpulseStateLearning[];
}

One of the most powerful features in Edge Impulse are the built-in deployment targets (under Deployment in the Studio), which let you create ready-to-go binaries for development boards, or custom libraries for a wide variety of targets that incorporate your trained impulse. You can also create custom deployment blocks for your organization. This lets developers quickly iterate on products without getting your embedded engineers involved, lets your customers build personalized firmware using their own data, or lets you create custom libraries.

In this tutorial you'll learn how to use custom deployment blocks to create a new deployment target, and how to make this target available in the Studio for all users in the organization.

Prerequisites

You'll need:

• The Edge Impulse CLI.

  • If you receive any warnings that's fine. Run edge-impulse-blocks afterwards to verify that the CLI was installed correctly.

Deployment blocks use Docker containers, a virtualization technique which lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):

• Docker desktop installed on your machine.

Then, create a new folder on your computer named custom-deploy-block.

1. Getting basic deployment info

When a user deploys with a custom deployment block two things happen:

1. A package is created that contains information about the deployment (like the sensors used, frequency of the data, etc.), any trained neural network in .tflite and SavedModel formats, the Edge Impulse SDK, and all DSP and ML blocks as C++ code.

2. This package is then consumed by the custom deployment block, which can incorporate it with a base firmware, or repackage it into a new library.

If you now go to the Deployment page, a new option appears under 'Create library':

Once you click Build you'll receive a ZIP file containing five items:

• deployment-metadata.json - this contains all information about the deployment, like the names of all classes, the frequency of the data, full impulse configuration, and quantization parameters. A specification can be found here: Deployment metadata spec.

• trained.tflite - if you have a neural network in the project this contains neural network in .tflite format. This network is already fully quantized if you choose the int8 optimization, otherwise this is the float32 model.

• trained.savedmodel.zip - if you have a neural network in the project this contains the full TensorFlow SavedModel. Note that we might update the TensorFlow version used to train these networks at any time, so rely on the compiled model or the TFLite file where possible.

• edge-impulse-sdk - a copy of the latest Inferencing SDK.

• model-parameters - impulse and block configuration in C++ format. Can be used by the SDK to quickly run your impulse.

• tflite-model - neural network as source code in a way that can be used by the SDK to quickly run your impulse.

Store the unzipped file under custom-deploy-block/input.

2. Building a new binary

With the basic information in place we can create a new deployment block. Here we'll build a standalone application that runs our impulse on Linux, very useful when running your impulse on a gateway or desktop computer. First, open a command prompt or terminal window, navigate to the custom-deploy-block folder (that you created under 1.), and run:

$ edge-impulse-blocks init

This will prompt you to log in, and enter the details for your block.

Next, we'll add the application. The base application can be found at edgeimpulse/example-standalone-inferencing.

1. Download the base application.

2. Unzip under custom-deploy-block/app.

To build this application we need to combine the application with the edge-impulse-sdk, model-parameters and tflite-model folder, and invoke the (already included) Makefile.

2.1 Creating a build script

To build the application we use Docker, a virtualization technique which lets developers package up an application with all dependencies in a single package. In this container we'll place the build tools required for this application, and scripts to combine the trained impulse with the base application.

First, let's create a small build script. As a parameter you'll receive --metadata which points to the deployment information. In here you'll also get information on the input and output folders where you need to read from and write to.

Create a new file called custom-deploy-block/build.py and add:

build.py

import argparse, json, os, shutil, zipfile, threading

# parse arguments (--metadata FILE is passed in)
parser = argparse.ArgumentParser(description='Custom deploy block demo')
parser.add_argument('--metadata', type=str)
args = parser.parse_args()

# load the metadata.json file
with open(args.metadata) as f:
    metadata = json.load(f)

# now we have two folders 'metadata.folders.input' - this is where all the SDKs etc are,
# and 'metadata.folders.output' - this is where we need to write our output
input_dir = metadata['folders']['input']
app_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'app')
output_dir = metadata['folders']['output']

print('Copying files to build directory...')

is_copying = True
def print_copy_progress():
    if (is_copying):
        threading.Timer(2.0, print_copy_progress).start()
        print("Still copying...")
print_copy_progress()

# create a build directory, the input / output folders are on network storage so might be very slow
build_dir = '/tmp/build'
if os.path.exists(build_dir):
    shutil.rmtree(build_dir)
os.makedirs(build_dir)

# copy in the data from both 'input' and 'app' folders
os.system('cp -r ' + input_dir + '/* ' + build_dir)
os.system('cp -r ' + app_dir + '/* ' + build_dir)

is_copying = False

print('Copying files to build directory OK')
print('')

print('Compiling application...')

is_compiling = True
def print_compile_progress():
    if (is_compiling):
        threading.Timer(2.0, print_compile_progress).start()
        print("Still compiling...")
print_compile_progress()

# then invoke Make
os.chdir(build_dir)
os.system('make -f Makefile.tflite')

is_compiling = False

print('Compiling application OK')

# ZIP the build folder up, and copy to output dir
if not os.path.exists(output_dir):
    os.makedirs(output_dir)
shutil.make_archive(os.path.join(output_dir, 'deploy'), 'zip', os.path.join(build_dir, 'build'))

Next, we need to create a Dockerfile, which contains all dependencies for the build. These include GNU Make, a compiler, and both the build script and the base application.

Create a new file called custom-deploy-block/Dockerfile and add:

Dockerfile

FROM ubuntu:18.04

WORKDIR /ei

# Install base dependencies
RUN apt update && apt install -y build-essential software-properties-common wget

# Install LLVM 9
RUN wget https://apt.llvm.org/llvm.sh && chmod +x llvm.sh && ./llvm.sh 9
RUN rm /usr/bin/gcc && rm /usr/bin/g++ && ln -s $(which clang-9) /usr/bin/gcc && ln -s $(which clang++-9) /usr/bin/g++

# Install Python 3.7
RUN apt install -y python3.7

# Copy the base application in
COPY app ./app

# Copy any scripts in that we have
COPY *.py ./

# This is the script our application should run (-u to disable buffering)
ENTRYPOINT [ "python3", "-u", "build.py" ]

2.2 Testing the build script with Docker

To test the build script we first build the container, then invoke it with the files from the input directory. Open a command prompt or terminal, navigate to the custom-deploy-block folder and:

1. Build the container:

   Copy

   $ docker build -t cdb-demo .

2. Invoke the build script - this mounts the current directory in the container under /home, and then passes the downloaded metadata script to the container:

   Copy

   $ docker run --rm -it -v $PWD:/home cdb-demo --metadata /home/input/deployment-metadata.json

3. Voila. You now have an output folder that contains a ZIP file. Unzip output/deploy.zip and now you have a standalone application which runs your impulse. If you run Linux you can invoke this application directly (grab some data from 'Live classification' for the features, see Running your impulse locally):

   Copy

   $ ./output/edge-impulse-standalone "RAW FEATURES HERE"

Or if you run Windows or macOS, you can use Docker to run this application:

```
$ docker run --rm -v $PWD/output:/home ubuntu:18.04 /home/edge-impulse-standalone "RAW FEATURES HERE"
```

3. Uploading the deployment block to Edge Impulse

With the deployment block ready you can make it available in Edge Impulse. Open a command prompt or terminal window, navigate to the folder you created earlier, and run:

$ edge-impulse-blocks push

This packages up your folder, sends it to Edge Impulse where it'll be built, and finally is added to your organization. The transformation block is now available in Edge Impulse under Deployment blocks. You can go here to set the logo, update the description, and set extra command line parameters.

Privileged mode

Deployment blocks do not have access to the internet by default. If you need this, or if you need to pull additional information from the project (e.g. access to DSP blocks) you can set the 'privileged' flag on a deployment block. This will enable outside internet access, and will pass in the project.apiKey parameter in the metadata (if a development API key is set) that you can use to authenticate with the Edge Impulse API.

4. Using the deployment block

The deployment block is automatically available for all organizational projects. Go to the Deployment page on a project, and you'll find a new section 'Custom targets'. Select your new deployment target and click Build.

And now you'll have a freshly built binary from your own deployment block!

5. Conclusion

Custom deployment blocks are a powerful tool for your organization. They let you build binaries for unreleased products, let you package up impulse as custom libraries, or can let your customers deploy to private targets (if you add an external collaborator to a project they'll have access to the blocks as well). Because the deployment blocks are integrated with your project, and hosted by Edge Impulse this lets everyone, from FAE to R&D developer, now iterate on on-device models without getting your embedded engineers involved.

You can also use custom deployment blocks with the other organizational features, and can use this to set up powerful pipelines automating data ingestion from your cloud services, transforming raw data into ML-suitable data, training new impulses and then deploying back to your device - either through the UI, or via the API. If you're interested in deployment blocks or any of the other enterprise features, let us know!