Data Sources

The data sources page is actually much more than just adding data from external sources. It let you create complete automated data pipelines so you can work on your active learning strategies.
From there, you can import datasets from existing cloud storage buckets, automate and schedule the imports, and, trigger actions such as explore and label your new data, retrain your model, automatically build a new deployment task and more.
Data sources
Add a data source
Click in + Add new data source and select where your data lives:
Add new data source
Click on Next, provide credentials:
Provide your credentials
Click on Verify credentials:
Automatically label your data
Here, you have several options to automatically label your data:
Infer from folder name
In the example above, the structure of the folder is the following:
1
.
2
├── cars
3
│   ├── cars.01741.jpg
4
│   ├── cars.01743.jpg
5
│   ├── cars.01745.jpg
6
│   ├── ... (400 items)
7
├── unknown
8
│   ├── unknown.test_2547.jpg
9
│   ├── unknown.test_2548.jpg
10
│   ├── unknown.test_2549.jpg
11
│   ├── ... (400 items)
12
└── unlabeled
13
    ├── cars.02066.jpg
14
    ├── cars.02067.jpg
15
    ├── cars.02068.jpg
16
    └── ... (14 items)
17
​
18
3 directories, 814 files
Copied!
The labels will be picked from the folder name and will be split between your training and testing set using the following ratio 80/20.
Note that the samples present in an unlabeled/ folder will be kept unlabeled in Edge Impulse Studio.
Alternatively, you can also organize your folder using the following structure to automatically split your dataset between training and testing sets:
1
.
2
├── testing
3
│   ├── cars
4
│   │   ├── cars.00012.jpg
5
│   │   ├── cars.00031.jpg
6
│   │   ├── cars.00035.jpg
7
│   │   └── ... (~150 items)
8
│   └── unknown
9
│       ├── unknown.test_1012.jpg
10
│       ├── unknown.test_1026.jpg
11
│       ├── unknown.test_1027.jpg
12
│       ├── ... (~150 items)
13
├── training
14
│   ├── cars
15
│   │   ├── cars.00006.jpg
16
│   │   ├── cars.00025.jpg
17
│   │   ├── cars.00065.jpg
18
│   │   └── ... (~600 items)
19
│   └── unknown
20
│       ├── unknown.test_1002.jpg
21
│       ├── unknown.test_1005.jpg
22
│       └── unknown.test_46.jpg
23
│       └── ... (~600 items)
24
└── unlabeled
25
    ├── cars.02066.jpg
26
    ├── cars.02067.jpg
27
    ├── cars.02068.jpg
28
    └── ... (14 items)
29
​
30
7 directories, 1512 files
Copied!
Infer from file name:
When using this option, only the file name is taken into account. The part before the first . will be used to set the label. E.g. cars.01741.jpg will set the label to cars.
Keep the data unlabeled:
All the data samples will be unlabeled, you will need to label them manually before using them.
Finally, click on Next, post-sync actions.
Trigger actions
From this view, you can automate several actions:
Recreate data explorer
The data explorer gives you a one-look view of your dataset, letting you quickly label unknown data. If you enable this you'll also get an email with a screenshot of the data explorer whenever there's new data.
Retrain model
If needed, will retrain your model with the same impulse. If you enable this you'll also get an email with the new validation and test set accuracy.
Note: You will need to have at least setup up and trained your project once.
Create new version
Store all data, configuration, intermediate results and final models.
Create new deployment
Builds a new library or binary with your updated model. Requires 'Retrain model' to also be enabled.
Run the pipeline
Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.
Run your pipeline
Run the pipeline from the UI
To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.
Run the pipeline from code
To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.
Note that you will need to create an API key to run the pipeline from code.
Run the pipeline from code
Schedule your pipeline jobs
By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.
Note that free users can only run the pipeline every 4 hours. If you are an enterprise customer, you can run this pipeline up to every minute.
Edit pipeline
Once the pipeline has successfully finish, you will receive an email like the following:
Email example containing the full results
Note that you can also define who can receive the email. The users have to be part of your project. See Dashboard -> Collaboration.
Webhooks
Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:
1
{
2
    "organizationId":XX,
3
    "pipelineId":XX,
4
    "pipelineName":"Import data from portal \"Data sources demo\"",
5
    "projectId":XXXXX,
6
    "success":true,
7
    "newItems":0,
8
    "newChecklistOK":0,
9
    "newChecklistFail":0
10
}
Copied!
Data sources webhooks
Edit your pipeline
As of today, if you want to update your pipeline, you need to edit the configuration json available in ⋮ -> Run pipeline from code.
Here is an example of what you can get if all the actions have been selected:
1
[
2
    {
3
        "name": "Fetch data from s3://data-pipeline/data-pipeline-example/infer-from-folder/",
4
        "builtinTransformationBlock": {
5
            "type": "s3-to-project",
6
            "endpoint": "https://s3.your-endpoint.com",
7
            "path": "s3://data-pipeline/data-pipeline-example/infer-from-folder/",
8
            "region": "fr-par",
9
            "accessKey": "XXXXX",
10
            "category": "split",
11
            "labelStrategy": "infer-from-folder-name",
12
            "secretKeyEncrypted": "xxxxxx"
13
        }
14
    },
15
    {
16
        "name": "Refresh data explorer",
17
        "builtinTransformationBlock": {
18
            "type": "project-action",
19
            "refreshDataExplorer": true
20
        }
21
    },
22
    {
23
        "name": "Retrain model",
24
        "builtinTransformationBlock": {
25
            "type": "project-action",
26
            "retrainModel": true
27
        }
28
    },
29
    {
30
        "name": "Create new version",
31
        "builtinTransformationBlock": {
32
            "type": "project-action",
33
            "createVersion": true
34
        }
35
    },
36
    {
37
        "name": "Create on-device deployment (C++ library)",
38
        "builtinTransformationBlock": {
39
            "type": "project-action",
40
            "buildBinary": "zip",
41
            "buildBinaryModelType": "int8"
42
        }
43
    }
44
]
Copied!
Free projects have only access to the above builtinTransformationBlock.
If you are part of an organization, you can use your custom transformation jobs in the pipeline. In your organization workspace, go to "Custom blocks -> Transformation" and select "Run job" on the job you want to add.
Transformation blocks
Select "Copy as pipeline step" and paste it to the configuration json file.
Copy

Edge Impulse Studio - Previous

Devices

Next - Edge Impulse Studio

Data Acquisition

Last modified 27d ago

Export as PDF

Copy link

Contents

Add a data source

Run the pipeline

Edit your pipeline