Remote Management
C++ Inference SDK Library
Edge Impulse API
Data acquisition format
To send data from any sensor into Edge Impulse you'll need to deliver the data in the Data acquisition format. This is a small specification that describes the type of data, the sensor data itself, and information about the device that generated the data. You can optionally sign this data so you have cryptographic proof on the origin and integrity of the data. There are end-to-end examples in C++, Python and Node.js here.
If you access to a fully-supported development board it will already send data in this format, and if you want to collect data from other devices the data forwarder will automatically convert and sign your data.
Data Acquisition Format specification
The Edge Impulse Data Acquisition format is an optimized binary format for time-series data. It allows cryptographic signing of the data when sampling is complete to prove authenticity of data. Data is encoded using CBOR or JSON and signed according to the JSON Web Signature specification.
Data can easily be generated by any language that has a CBOR or JSON library available, but there are examples available for C, C++, Node.js and Python.
Example in CBOR diagnostic notation:
1
{
2
"protected": {
3
"ver": "v1",
4
"alg": "HS256",
5
"iat": 1564128599
6
},
7
"signature": "b0ee0572a1984b93b6bc56e6576e2cbbd6bccd65d0c356e26b31bbc9a48210c6",
8
"payload": {
9
"device_name": "ac:87:a3:0a:2d:1b",
10
"device_type": "DISCO-L475VG-IOT01A",
11
"interval_ms": 10,
12
"sensors": [
13
{ "name": "accX", "units": "m/s2" },
14
{ "name": "accY", "units": "m/s2" },
15
{ "name": "accZ", "units": "m/s2" }
16
],
17
"values": [
18
[ -9.81, 0.03, 1.21 ],
19
[ -9.83, 0.04, 1.27 ],
20
[ -9.12, 0.03, 1.23 ],
21
[ -9.14, 0.01, 1.25 ]
22
]
23
}
24
}
Copied!
Fields
protected- information about the signature format.ver- alwaysv1(required).alg- the algorithm used to sign this file. EitherHS256(HMAC-SHA256) ornone(required).iat- date when the file was created in seconds since epoch. Only set this when the device creating the file has an accurate clock (optional).
payload- sensor data.device_name- unique identifier for this device. Only set this when the device has a globally unique identifier (e.g. MAC address). If this field is set the device shows up on the 'Devices' page in the studio (optional).device_type- device type, for example the exact model of the device. Should be the same for all similar devices (required).interval_ms- the frequency of the data in this file (in milliseconds). E.g. for 100Hz fill in10(new data every 10 ms.). You can use a float here if you need the precision (required). If you have sensors with different sampling frequencies, you should upscale to the highest frequency to keep the finest granularity. Example: sensor A is 100 Hz, sensor B is 5 Hz. Upscale sensor B to 100 Hz by duplicating each value 20 times (100/5). You could also smooth values over between samples.sensors- array with sensor axes.name- name of the axis.
values- array of sensor values. One array item per interval, and as many items in this array as there are sensor axes. If you have a single sensor, you are allowed to flatten this array to save space. An array of string values can be used to refer to binary attachments sent to the ingestion service using multipart requests (see Binary payloads and Binary attachments).
Signing data
Files can be signed to establish a trust chain between device and Edge Impulse. Because the
iat and device_name fields are included in the signed file you can validate authenticity of the data, which device created the data, and when the data was captured. Currently data can be signed with a symmetric HMAC key. You can create HMAC keys under 'Dashboard' in the studio.HMAC SHA256
To sign data using HMAC SHA256:
- 1.Set the
signaturefield to0000000000000000000000000000000000000000000000000000000000000000(64 times ASCII0). - 2.Create the full CBOR or JSON object.
- 3.Use your language's crypto module to sign the object with HMAC-SHA256. This should give you a 32 byte signature.
- 4.Replace the empty signature with the HEX encoded signature (64 characters ASCII).
The C, C++, Node.js and Python SDKs all support signing data using HMAC SHA256. See the examples section.
Binary notation (CBOR)
1
a3 69 70 72 6f 74 65 63 74 65 64 a3 63 76 65 72 62 76 31 63 61 6c 67 65 48 53 32 35 36 63 69 61 74 1a 5d 3a dd 3d 69 73 69 67 6e 61 74 75 72 65 78 40 62 30 65 65 30 35 37 32 61 31 39 38 34 62 39 33 62 36 62 63 35 36 65 36 35 37 36 65 32 63 62 62 64 36 62 63 63 64 36 35 64 30 63 33 35 36 65 32 36 62 33 31 62 62 63 39 61 34 38 32 31 30 63 36 67 70 61 79 6c 6f 61 64 a5 6b 64 65 76 69 63 65 5f 6e 61 6d 65 71 61 63 3a 38 37 3a 61 33 3a 30 61 3a 32 64 3a 31 62 6b 64 65 76 69 63 65 5f 74 79 70 65 73 44 49 53 43 4f 2d 4c 34 37 35 56 47 2d 49 4f 54 30 31 41 6b 69 6e 74 65 72 76 61 6c 5f 6d 73 0a 67 73 65 6e 73 6f 72 73 83 a2 64 6e 61 6d 65 64 61 63 63 58 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 59 65 75 6e 69 74 73 64 6d 2f 73 32 a2 64 6e 61 6d 65 64 61 63 63 5a 65 75 6e 69 74 73 64 6d 2f 73 32 66 76 61 6c 75 65 73 9f 83 fa c1 1c f5 c3 fa 3c f5 c2 8f fa 3f 9a e1 48 83 fa c1 1d 47 ae fa 3d 23 d7 0a fa 3f a3 d7 0a 83 fa c1 11 eb 85 fa 3c f5 c2 8f fa 3f 9d 70 a4 83 fa c1 12 3d 71 fa 3c 23 d7 0a f9 3d 00 ff
Copied!
Examples
C/C++
Node.js
Python
1
// First, add the Edge Impulse C Ingestion SDK to your project.
2
// https://github.com/edgeimpulse/ingestion-sdk-c
3
// See 'C SDK Usage Guide' for more information and porting information
4
​
5
#include "sensor_aq.h"
6
#include "sensor_aq_mbedtls_hs256.h"
7
​
8
const char *hmac_key = "fed53116f20684c067774ebf9e7bcbdc";
9
​
10
int main() {
11
// The sensor format supports signing the data, set up a signing context
12
sensor_aq_signing_ctx_t signing_ctx;
13
​
14
// We'll use HMAC SHA256 signatures, which can be created through Mbed TLS
15
// If you use a different crypto library you can implement your own context
16
sensor_aq_mbedtls_hs256_ctx_t hs_ctx;
17
// Set up the context, the last argument is the HMAC key
18
sensor_aq_init_mbedtls_hs256_context(&signing_ctx, &hs_ctx, hmac_key);
19
​
20
// Set up the sensor acquisition basic context
21
sensor_aq_ctx ctx = {
22
// We need a single buffer. The library does not require any dynamic allocation (but your TLS library might)
23
{ (unsigned char*)malloc(1024), 1024 },
24
​
25
// Pass in the signing context
26
&signing_ctx,
27
​
28
// And pointers to fwrite and fseek - note that these are pluggable so you can work with them on
29
// non-POSIX systems too. Just override the EI_SENSOR_AQ_STREAM macro to your custom file type.
30
&fwrite,
31
&fseek,
32
// if you set the time function this will add 'iat' (issued at) field to the header with the current time
33
// if you don't include it, this will be omitted
34
&time
35
};
36
​
37
// Payload header
38
sensor_aq_payload_info payload = {
39
// Unique device ID (optional), set this to e.g. MAC address or device EUI **if** your device has one
40
"ac:87:a3:0a:2d:1b",
41
// Device type (required), use the same device type for similar devices
42
"DISCO-L475VG-IOT01A",
43
// How often new data is sampled in ms. (100Hz = every 10 ms.)
44
10,
45
// The axes which you'll use. The units field needs to comply to SenML units (see https://www.iana.org/assignments/senml/senml.xhtml)
46
{ { "accX", "m/s2" }, { "accY", "m/s2" }, { "accZ", "m/s2" } }
47
};
48
​
49
// Place to write our data.
50
// The library streams data, and does not cache everything in buffers
51
FILE *file = fopen("test/encoded.cbor", "w+");
52
​
53
// Initialize the context, this verifies that all requirements are present
54
// it also writes the initial CBOR structure
55
int res;
56
res = sensor_aq_init(&ctx, &payload, file, false);
57
if (res != AQ_OK) {
58
printf("sensor_aq_init failed (%d)\n", res);
59
return 1;
60
}
61
​
62
// Periodically call `sensor_aq_add_data` (every 10 ms. in this example) to append data
63
float values[][3] = {
64
{ -9.81, 0.03, 1.21 },
65
{ -9.83, 0.04, 1.28 },
66
{ -9.12, 0.03, 1.23 },
67
{ -9.14, 0.01, 1.25 }
68
};
69
for (size_t ix = 0; ix < sizeof(values) / sizeof(values[0]); ix++) {
70
res = sensor_aq_add_data(&ctx, values[ix], 3);
71
if (res != AQ_OK) {
72
printf("sensor_aq_add_data failed (%d)\n", res);
73
return 1;
74
}
75
}
76
​
77
// When you're done call sensor_aq_finish - this will calculate the finalized signature and close the CBOR file
78
res = sensor_aq_finish(&ctx);
79
if (res != AQ_OK) {
80
printf("sensor_aq_finish failed (%d)\n", res);
81
return 1;
82
}
83
​
84
// Use the HTTP libraries available for your platform to upload
85
// test/encoded.cbor
86
// to
87
// https://ingestion.edgeimpulse.com/api/training/data
88
}
Copied!
1
const fs = require('fs');
2
const crypto = require('crypto');
3
const Path = require('path');
4
const request = require('request');
5
​
6
const hmac_key = "fed53116f20684c067774ebf9e7bcbdc";
7
const API_KEY = "ei_fd83...";
8
​
9
// empty signature (all zeros). HS256 gives 32 byte signature, and we encode in hex, so we need 64 characters here
10
let emptySignature = Array(64).fill('0').join('');
11
​
12
let data = {
13
protected: {
14
ver: "v1",
15
alg: "HS256",
16
iat: Math.floor(Date.now() / 1000) // epoch time, seconds since 1970
17
},
18
signature: emptySignature,
19
payload: {
20
device_name: "ac:87:a3:0a:2d:1b",
21
device_type: "DISCO-L475VG-IOT01A",
22
interval_ms: 10,
23
sensors: [
24
{ name: "accX", units: "m/s2" },
25
{ name: "accY", units: "m/s2" },
26
{ name: "accZ", units: "m/s2" }
27
],
28
values: [
29
[ -9.81, 0.03, 1.21 ],
30
[ -9.83, 0.04, 1.27 ],
31
[ -9.12, 0.03, 1.23 ],
32
[ -9.14, 0.01, 1.25 ]
33
]
34
}
35
};
36
​
37
let encoded = JSON.stringify(data);
38
​
39
// now calculate the HMAC and fill in the signature
40
let hmac = crypto.createHmac('sha256', hmac_key);
41
hmac.update(encoded);
42
let signature = hmac.digest().toString('hex');
43
​
44
// update the signature in the message and re-encode
45
data.signature = signature;
46
encoded = JSON.stringify(data);
47
​
48
// now upload the buffer to Edge Impulse
49
request.post('https://ingestion.edgeimpulse.com/api/training/data', {
50
headers: {
51
'x-api-key': API_KEY,
52
'x-file-name': 'test01',
53
'Content-Type': 'application/json'
54
},
55
body: encoded,
56
encoding: 'binary'
57
}, function (err, response, body) {
58
if (err) return console.error('Request failed', err);
59
​
60
console.log('Uploaded file to Edge Impulse', response.statusCode, body);
61
});
Copied!
1
# First, install the dependencies via:
2
# $ pip3 install requests
3
​
4
import json
5
import time, hmac, hashlib
6
import requests
7
​
8
HMAC_KEY = "fed53116f20684c067774ebf9e7bcbdc"
9
API_KEY = "ei_fd83..."
10
​
11
# empty signature (all zeros). HS256 gives 32 byte signature, and we encode in hex, so we need 64 characters here
12
emptySignature = ''.join(['0'] * 64)
13
​
14
data = {
15
"protected": {
16
"ver": "v1",
17
"alg": "HS256",
18
"iat": time.time() # epoch time, seconds since 1970
19
},
20
"signature": emptySignature,
21
"payload": {
22
"device_name": "ac:87:a3:0a:2d:1b",
23
"device_type": "DISCO-L475VG-IOT01A",
24
"interval_ms": 10,
25
"sensors": [
26
{ "name": "accX", "units": "m/s2" },
27
{ "name": "accY", "units": "m/s2" },
28
{ "name": "accZ", "units": "m/s2" }
29
],
30
"values": [
31
[ -9.81, 0.03, 1.21 ],
32
[ -9.83, 0.04, 1.27 ],
33
[ -9.12, 0.03, 1.23 ],
34
[ -9.14, 0.01, 1.25 ]
35
]
36
}
37
}
38
​
39
# encode in JSON
40
encoded = json.dumps(data)
41
​
42
# sign message
43
signature = hmac.new(bytes(HMAC_KEY, 'utf-8'), msg = encoded.encode('utf-8'), digestmod = hashlib.sha256).hexdigest()
44
​
45
# set the signature again in the message, and encode again
46
data['signature'] = signature
47
encoded = json.dumps(data)
48
​
49
# and upload the file
50
res = requests.post(url='https://ingestion.edgeimpulse.com/api/training/data',
51
data=encoded,
52
headers={
53
'Content-Type': 'application/json',
54
'x-file-name': 'idle01',
55
'x-api-key': API_KEY
56
})
57
if (res.status_code == 200):
58
print('Uploaded file to Edge Impulse', res.status_code, res.content)
59
else:
60
print('Failed to upload file to Edge Impulse', res.status_code, res.content)
Copied!
What about applying labels to your samples? You can use the optional
x-label header parameter when creating your request. See the header parameters available in the ingestion API here.Binary payloads (e.g. for audio)
For some payloads, like audio data, it's not feasible to convert the raw sensor data into CBOR values while capturing data on the device. For these cases you can send a binary payload. The same header structure as above applies, but instead of writing the values in the
values CBOR field you can append a binary array at the end of the file (in uint8, uint16, uint32, int8, int16, int32 or float32 format). To do so:- 1.Limit your data to a single sensor.
- 2.Set the value of the
valuesfield to:[ "Ref-Binary-i16" ](so a string within an array, wherei16indicatesint16, alternatively you can useu16foruint16orf32forfloat32). - 3.Write the CBOR header. Note that this needs to be a valid CBOR structure, and that it requires termination with
0xFF. The easiest to achieve this is by writing thevaluesfield as an indefinite length array (which needs to be terminated by0xFF. - 4.Write your payload (little endian).
- If you need to align your writes the easiest is to pad the
Ref-Binary-i16string with spaces on the right.
- 5.Update the signature with the signature for the full file, including header and payload.
- 6.Upload the data to the ingestion service with the
Content-Type: application/octet-streamheader.
Here is an example of a valid audio sample that includes a binary payload:
1
a36970726f746563746564a26376657262763163616c67654853323536697369676e6174757265784038346432316234343665323831383561356336303761396437656134636336333735356137363261666164643036343334613136353766623433386237656630677061796c6f6164a56b6465766963655f6e616d657143343a37463a35313a38443a31383a34416b6465766963655f7479706573444953434f5f4c34373556475f494f543031416b696e74657276616c5f6d73f92c006773656e736f727381a2646e616d6565617564696f65756e697473637761766676616c7565739f6e5265662d42494e4152592d693136fff918fa18e718de18e218dd18e118e218f218f018f918f418e918ee18f618f918fc180919101915191119fe18f6180b190e190a19021912191419fc18f218f718
Copied!
The header stops at byte
ff (you can copy the beginning of the data up until the ff into cbor.me to validate) and the payload starts at f918fa.Binary attachments (e.g. for images)
For non-time-series data - currently only implemented for images - you can send binary attachments to the ingestion service. Here the same header structure is used as above, but instead of writing the values in the
values CBOR field, you reference the content type, the size, and the signed hash (with the same HMAC key) of your attachment. You then need to send both the header and the actual file to the Ingestion service via a multipart/form request (see 'Multipart requests' section).An end-to-end example that demonstrates how to create a valid message, and post it to the ingestion service can be found here: example-ingestion-jpg.
Note: Including more than 1 binary attachment is currently not supported by the Edge Impulse Studio, such messages won't be rejected by the ingestion service though, these attachments will merely be 'invisible' from the studio.
Last modified 1mo ago