Running Nvidia TAO models on the Renesas RA8D1

The Nvidia TAO model zoo offers a large variety of powerful pre-trained models that can help you get started on and make good progress on your projects in a very short period of time. With Edge Impulse's partnership with Nvidia, these models are now available for you to be used in your projects. The Renesas RA8D1 evaluation kit, equipped with the most advanced Cortex-M85 microcontroller and ARM Helium technology, is a powerhouse for edge AI applications. This tutorial will walk you through the steps of harnessing the combined power of Nvidia TAO and the Cortex-M85, by running your TAO based projects on the Renesas RA8D1.

Prerequisites

To follow this tutorial you will need:

• a Renesas EK-RA8D1 evaluation kit. The kit can be purchased here.

• the Renesas e2studio IDE and coding tool. This can be downloaded here.

• an Edge Impulse project using an Nvidia TAO training block. This step is detailed in this document.

Creating a project with Nvidia TAO blocks

To follow this tutorial, you will need to build an object detection or image classification project. We will create an object detection project using the following steps:

1. Create a new project in Edge Impulse.

2. Make sure to set your labelling method to 'Bounding boxes (object detection)'.

3. Collect and prepare your dataset as detailed in object detection. In this tutorial, we will be creating a dataset to detect bottles vs. cans.

4. Create your impulse and add an 'NVIDIA TAO ...' block to it. Some blocks have a requirement for the input images to be a multiple of 32. If that's the case for the block you chose, resize the input image to a correct size e.g. 96x96 or 160x160.

   

   Select model
5. Under NVIDIA TAO... block, select between various parameters. Each TAO pre-trained block supports several backbones trained on subsets of Google OpenImages dataset. In total there are 88 object detection architectures. For this tutorial, we will choose the MobileNet v2 3x224x224 backbone and leave the remaining settings to their default values.

   

   Select model

There are several pre-trained 3x224x224 backbones from the NVIDIA TAO catalog, and others trained by Edge Impulse on ImageNet.

1. Click on 'Start training' to train the model.

Deploying on the RA8D1

Once the model has finished training, you will be ready to deploy it to your device. For non-Nvidia TAO models, the process to deploy on the RA8 is the standard method described here. However, applications using TAO models require more space to store the model and application, which is why we need to manually allocate the storage location in the RA8 firmware before flashing. This is a multi-step process described below.

1. Download the RA8 firmware from the repository here.

2. Open this firmware in an IDE to access the build settings. We recommend using the renesas e2studio IDE mentioned in the prerequisites. Simply, open the IDE and choose the option to import as preexisting project from folder or archive. Once imported, navigate to project > settings > C/C++ build > settings > tool settings.

3. In the 'Tool Settings' tab, navigate to GNU Arm Cross C++ Compiler > Preprocessor. Under these settings, under 'Defined symbols (-D)', add EI_CLASSIFIER_ALLOCATION_STATIC as an additional defined symbol. This is also shown below:

   

   Add symbol to project properties
4. Hit 'Apply and Close' and build the project.

5. Now go back to the project created in studio in the section 'Creating a project with Nvidia TAO blocks'. Download the project as a C++ library instead of an RA8 deployment.

   

   Download as C++ library
6. After downloading the project as a C++ library, navigate to tflite-model/tflite_learn_##_compiled.cpp. The '##' will be a number that will vary for each download.

7. Within this file, find the following block of code (usually around line 74):

   #if defined(EI_CLASSIFIER_ALLOCATION_STATIC)
   uint8_t tensor_arena[kTensorArenaSize] ALIGN(16);

   and replace it with the following:

   #if defined(EI_CLASSIFIER_ALLOCATION_STATIC)
   uint8_t tensor_arena[kTensorArenaSize] ALIGN(16) __attribute__((section(".sdram")));

   This will force the tensor arena, the area where the model is stored, to be created in the external storage which has 64MB SDRAM capacity, which is more than the internal 2MB Code Flash, 1MB SRAM capacity.

8. After making these changes, navigate back to the firmware downloaded in step 1. The file structure in this repository is as follows:

   /home
   │
   ├── flash-script
   │   ├── flash_linux.sh
   │   ...
   ├── script
   │   └── fsp.ld
   ├── src
   │   ├── edge-impulse-sdk
   │   ├── firmware-sdk
   │   ├── graphic
   │   ├── inference
   │   ├── ingestion-sdk-platform
   │   ├── ingestion-sdk
   │   ├── model
   │       └── model-parameters
   │       └── tflite-model
   │   └── peripheral
   ...

9. Here, replace the folders model/model-parameters and model/tflite-model, by the same folders downloaded from the studio as a C++ library. This will overwrite the standard firmware, with the one we modified to accommodate the Nvidia TAO model.

10. Follow the build instructions from the firmware repository to build and flash the firmware to your RA8.

Now you are ready to use your Nvidia TAO based model on the Renesas RA8 device!

Troubleshooting

Failed to install the Renesas e2studio on MacOS

Sometimes after installing the e2studio application on your MacBook, you might see the error "E2studio.app" is damaged and cannot be opened when trying to open the application. You can resolve this problem by using the following terminal command to mark the it as “valid” after extracting it from the archive file:

xattr -d com.apple.quarantine /path/to/E2studio.app

making sure to replace /path/to/E2studio.app with the actual location of your e2studio application. One easy way of doing this is to drag the E2studio.app icon from your Finder window into the Terminal window, right after typing the initial part of the command. After this, you should be able to use the application without any issues.