Data sources
Last updated
Last updated
The data sources page is much more than just adding data from external sources. It lets you create complete automated data pipelines so you can work on your active learning strategies.
From there, you can import datasets from existing cloud storage buckets, automate and schedule the imports, and, trigger actions such as explore and label your new data, retrain your model, automatically build a new deployment task and more.
Click in + Add new data source and select where your data lives:
You can either use:
AWS S3 buckets
Google Cloud Storage
Any S3-compatible bucket
Don't import data (if you just need to create a pipeline)
Click on Next, provide credentials:
Click on Verify credentials:
Here, you have several options to automatically label your data:
In the example above, the structure of the folder is the following:
The labels will be picked from the folder name and will be split between your training and testing set using the following ratio 80/20.
Alternatively, you can also organize your folder using the following structure to automatically split your dataset between training and testing sets:
When using this option, only the file name is taken into account. The part before the first . will be used to set the label. E.g. cars.01741.jpg will set the label to cars.
All the data samples will be unlabeled, you will need to label them manually before using them.
Finally, click on Next, post-sync actions.
From this view, you can automate several actions:
Recreate data explorer
Retrain model
If needed, will retrain your model with the same impulse. If you enable this you'll also get an email with the new validation and test set accuracy.
Note: You will need to have trained your project at least once.
Create new version
Store all data, configuration, intermediate results and final models.
Create new deployment
Builds a new library or binary with your updated model. Requires 'Retrain model' to also be enabled.
Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.
To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.
To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.
By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.
Once the pipeline has successfully finish, you will receive an email like the following:
Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:
As of today, if you want to update your pipeline, you need to edit the configuration json available in ⋮ -> Run pipeline from code.
Here is an example of what you can get if all the actions have been selected:
Free projects have only access to the above builtinTransformationBlock.
Select Copy as pipeline step and paste it to the configuration json file.
(enterprise feature)
(enterprise feature)
The gives you a one-look view of your dataset, letting you quickly label unknown data. If you enable this you'll also get an email with a screenshot of the data explorer whenever there's new data.
You can also define who can receive the email. The users have to be part of your project. See: .
If you are part of an , you can use your custom transformation jobs in the pipeline. In your organization workspace, go to Custom blocks -> Transformation and select Run job on the job you want to add.
