Upload portals

Upload portals are a secure way to let external parties upload data to your datasets. Through an upload portal they get an easy user interface to add data, but they have no access to the content of the dataset, nor can they delete any files. Data that is uploaded through the portal can be stored on-premise or in your own cloud infrastructure.

In this tutorial we'll set up an upload portal, show you how to add new data, and how to show this data in Edge Impulse for further processing.

Only available for enterprise customers

Organizational features are only available for enterprise customers. View our pricing for more information.

1. Configuring a storage bucket

Data is stored in storage buckets, which can either be hosted by Edge Impulse, or in your own infrastructure.

2. Creating an upload portal

With your storage bucket configured you're ready to set up your first upload portal. In your organization go to Data > Upload portals and choose Create new upload portal. Here, select a name, a description, the storage bucket, and a path in the storage bucket.

Creating an upload portal

Note: You'll need to enable CORS headers on the bucket. If these are not configured you'll get prompted with instructions. Talk to your user success engineer (when your data is hosted by Edge Impulse), or your system administrator to configure this.

After your portal is created a link is shown. This link contains an authentication token, and can be shared directly with the third party.

An active upload portal

Click the link to open the portal. If you ever forget the link: no worries. Click the ⋮ next to your portal, and choose View portal.

3. Uploading data to the portal

An upload portal with two folders.

To upload data you can now drag & drop files or folders to the drop zone on the right, or use Create new folder to first create a folder structure. There's no limit to the amount of files you can upload here, and all files are hashed, so if you upload a file that's already present the file will be skipped.

Note: Files with the same name but with a different hash are overwritten.

4. Using your portal in transformation blocks / research data

If you want to process data in a portal as part of a Research Pipeline you can either:

1. Mount the portal directly into a transformation block via Custom blocks > Transformation blocks > Edit block, and select the portal under mount points.

2. Mount the bucket that the portal is in, as a transformation block. This will also give you access to all other data in the bucket, very useful if you need to sync other data (see Synchronizing research data with a bucket).

5. Adding the data to your project

If the data in your portal is already in the right format you can also directly import the uploaded data to your project. In your project view, go to Data Sources, **** select 'Upload portal' and follow the steps of the wizard:

Data Sources - Upload portal method

6. Recap

If you need a secure way for external parties to contribute data to your datasets then upload portals are the way to go. They offer a friendly user interface, upload data directly into your storage buckets, and give you an easy way to use the data directly in Edge Impulse. 🚀

Any questions, or interested in the enterprise version of Edge Impulse? Contact us for more information.

Appendix A: Uploading data programmatically

Here's a Python script which lets you upload data to a portal. It first creates a signed URL to upload data to the bucket, then uploads data directly.

import requests
import json
import os
import hashlib

PORTAL_TOKEN = os.environ.get('EI_PORTAL_TOKEN')
PORTAL_ID = os.environ.get('EI_PORTAL_ID')

if not PORTAL_TOKEN:
    print('Missing EI_PORTAL_TOKEN environmental variable.')
    print('Go to a portal, and copy the part after "?token=" .')
    print('Then run:')
    print('    export EI_PORTAL_TOKEN=ec61e...')
    print('')
    print('You can add the line above to your ~/.bashrc or ~/.zshrc file to automatically load the token in the future.')
    exit(1)

if not PORTAL_ID:
    print('Missing EI_PORTAL_ID environmental variable.')
    print('Go to a portal, open the browser console, and look for "portalId" in the "Hello world from Edge Impulse" object to find it.')
    print('Then run:')
    print('    export EI_PORTAL_ID=122')
    print('')
    print('You can add the line above to your ~/.bashrc or ~/.zshrc file to automatically load the token in the future.')
    exit(1)

def get_file_hash(path):
    with open(path, 'rb') as f:
        return hashlib.md5(f.read()).hexdigest()

def create_upload_link(file_name_in_portal, path):
    url = "https://studio.edgeimpulse.com/v1/api/portals/" + PORTAL_ID + "/upload-link"

    payload = json.dumps({
        'fileName': file_name_in_portal,
        "fileSize": os.path.getsize(path),
        "fileHash": get_file_hash(path)
    })
    headers = {
        'accept': "application/json",
        'content-type': "application/json",
        'x-token': PORTAL_TOKEN
    }

    response = requests.request("POST", url, data=payload, headers=headers)

    if (response.status_code != 200):
        raise Exception('status code was not 200, but ' + str(response.status_code) + ' - ' + response.text)

    j = response.json()
    if (not j['success']):
        raise Exception('api request did not succeed ' + str(response.status_code) + ' - ' + response.text)

    return j['url']

def upload_file_to_s3(signed_url, path):
    with open(path, 'rb') as f:
        response = requests.request("PUT", signed_url, data=f, headers={})

        if (response.status_code != 200):
            raise Exception('status code was not 200, but ' + str(response.status_code) + ' - ' + response.text)

def upload_file_to_portal(file_name_in_portal, path):
    print('Uploading', file_name_in_portal + '...')
    link = create_upload_link(file_name_in_portal, path)
    upload_file_to_s3(link, path)
    print('Uploading', file_name_in_portal, 'OK')
    print('')

# this is how you upload files to a portal using the Edge Impulse API
# first argument is the path in the portal, second is the location of the file
upload_file_to_portal('test.jpg', '/Users/janjongboom/Downloads/test.jpg')

# uploading to a subdirectory
upload_file_to_portal('flowers/daisy.jpg', '/Users/janjongboom/Downloads/daisy-resized.jpg')

print('Done!')