Uploader
The uploader signs local files and uploads them to the ingestion service. This is useful to upload existing data samples and entire datasets, or to migrate data between Edge Impulse instances.
The uploader currently handles these types of files:
.cbor- Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them..json- Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them..csv- Files in the Edge Impulse Comma Separated Values (CSV) format. If you have configured the "CSV wizard", the settings will be used to parse your CSV files..wav- Lossless audio files. It's recommended to use the same frequency for all files in your data set, as signal processing output might be dependent on the frequency..jpgand.png- Image files. It's recommended to use the same ratio for all files in your data set..mp4and.avi- Video file. You can then from the studio split this video file into images at a configurable frame per second.
The uploader currently handles these types of dataset annotation formats:
Upload data from the studio
You can also upload data directly from the studio, see Studio uploader. Go to the Data acquisition page, and click the 'upload' icon. You can select files or folders, the category and the label directly from here.
Uploading via the CLI
You can upload files via the Edge Impulse CLI via:
edge-impulse-uploader path/to/a/file.wavYou can upload multiple files in one go via glob patterns:
# Upload any wave files in a single directory
edge-impulse-uploader path/to/many/*.wav
# Another example, upload any wave in any subdirectory
edge-impulse-uploader **/*.wavOr by specifying the directory:
edge-impulse-uploader --directory {path}The first time you'll be prompted for a server, and your login credentials (see Edge Impulse Daemon for more information).
Category
Files are automatically uploaded to the training category, but you can override the category with the --category option. E.g.:
edge-impulse-uploader --category testing path/to/a/file.wavOr set the category to split to automatically split data between training and testing sets (recommended for a balanced dataset). This is based on the hash of the file, so this is a deterministic process.
Labeling
A label is automatically inferred from the file name, see the Ingestion service documentation. You can override this with the --label option. E.g.:
edge-impulse-uploader --label noise path/to/a/file.wavCustom labeling and metadata
When a labeling method is not provided, the labels are automatically inferred from the filename through the following regex: ^[a-zA-Z0-9\s-_]+. For example: idle.01 will yield the label idle.
Thus, if you want to use labels (string values) containing float values (e.g. "0.01", "5.02", etc...), automatic labeling won't work.
To bypass this limitation, you can make a JSON file containing your dataset files' info. We also support adding metadata to your samples:
info.labels
{
"version": 1,
"files": [{
"path": "path1.csv",
"category": "split",
"label": { "type": "label", "label": "0.5" },
"metadata": {
"site_collected": "Amsterdam"
}
}, {
"path": "path2.csv",
"category": "split",
"label": { "type": "label", "label": "0.7" },
"metadata": {
"site_collected": "Paris"
}
}]
}Metadata field is optional
To upload unlabeled data use
"label": { "type": "unlabeled" }To upload multi-label data use
"label": {
"type": "multi-label",
"labels": [
{
"label": "noise",
"startIndex": 0,
"endIndex": 5000
},
{
"label": "nominal_mode",
"startIndex": 5001
"endIndex": 60000
},
{
"label": "defect",
"startIndex": 60001
"endIndex": 60200
}
],And then use:
edge-impulse-uploader --info-file info.labelsOther dataset annotation formats
(available since Edge Impulse CLI v1.21)
You can upload directories of data in a range of different formats:
edge-impulse-uploader --directory {path}By default, we try to automatically detect your dataset annotation format from the supported ones. If we cannot detect it, the uploader will output the list of formats. You can then use:
edge-impulse-uploader --directory {path} --dataset-format {name}Clearing configuration
To clear the configuration, run:
edge-impulse-uploader --cleanThis resets the uploader configuration and will prompt you to log in again.
API Key
You can use an API key to authenticate with:
edge-impulse-uploader --api-key ei_...Note that this resets the uploader configuration and automatically configures the uploader's account and project.
Bounding boxes
If you want to upload data for object detection, the uploader can label the data for you as it uploads it. To do this, all you need is to create a bounding_boxes.labels file in the same folder as your image files. The contents of this file are formatted as JSON with the following structure:
{
"version": 1,
"type": "bounding-box-labels",
"boundingBoxes": {
"mypicture.jpg": [{
"label": "jan",
"x": 119,
"y": 64,
"width": 206,
"height": 291
}, {
"label": "sami",
"x": 377,
"y": 270,
"width": 158,
"height": 165
}]
}
}You can have multiple keys under the boundingBoxes object, one for each file name. If you have data in multiple folders, you can create a bounding_boxes.labels in each folder.
Upload data from OpenMV datasets
The uploader data in the OpenMV dataset format. Pass in the option --format-openmv and pass the folder of your dataset in to automatically upload data. Data is automatically split between testing and training sets. E.g.:
$ edge-impulse-uploader --format-openmv path/to/your-openmv-datasetOther options
--silent- omits information on startup. Still prints progress information.--dev- lists development servers, use in conjunction with--clean.--hmac-key <key>- set the HMAC key, only used for files that need to be signed such aswavfiles.--concurrency <count>- number of files to uploaded in parallel (default: 20).--progress-start-ix <index>- when set, the progress index will start at this number. Useful to split up large uploads in multiple commands while the user still sees this as one command.--progress-end-ix <index>- when set, the progress index will end at this number. Useful to split up large uploads in multiple commands while the user still sees this as one command.--progress-interval <interval>- when set, the uploader will not print an update for every line, but everyintervalperiod (in ms.).--allow-duplicates- to avoid pollution of your dataset with duplicates, the hash of a file is checked before uploading against known files in your dataset. Enable this flag to skip this check.
Uploading large datasets
When using command line wildcards to upload large datasets you may encounter an error similar to this one:
edge-impulse-uploader *.wav
zsh: argument list too long: edge-impulse-uploaderThis happens if the number of .wav files exceeds the total number of arguments allowed for a single command on your shell. You can easily work around this shell limitation by using the find command to call the uploader for manageable batches of files:
find . -name "*.wav" -print0 | xargs -0 edge-impulse-uploaderYou can include any necessary flags by appending them to the xargs portion, for example if you wish to specify a category:
find training -name "*.wav" -print0 | xargs -0 edge-impulse-uploader --category trainingLast updated
Was this helpful?