Use this file to discover all available pages before exploring further.
The CSV Wizard allows users with larger or more complex datasets to easily upload their data without having to worry about converting it to the data acquisition format by creating a configuration file that will be applied to all future data uploads.The CSV Wizard supports uploading reference CSV, TXT, and Parquet ( .csv, .txt, and .parquet) files to create a configuration file, or uploading an existing JSON configuration file, making it a versatile tool for data ingestion.
The CSV Wizard creates a configuration for future data uploadsGoing through the CSV Wizard creates a configuration file that will be applied to all future data uploads. After creating the configuration, you must still upload your data. You can upload as many data files as needed and the configuration will be automatically applied.
To access the CSV Wizard, navigate to the Data acquisition view of your Edge Impulse project and click on the CSV Wizard tab.We will take a look at some sample data to see how the CSV Wizard works. A very basic CSV file was created to walk through the steps below. You can follow along with your own data or use the sample data provided:
Choose a CSV, TXT, or Parquet file that is representative of your data, then click the Upload file button. The file will be automatically analyzed and the results will be displayed in the next step.Alternatively, if you have previously gone through the CSV Wizard and already have a JSON configuration file you would like to reuse, you can upload it in this step to apply the configuration to your project.
The file you upload is processed and presented to ensure that it looks accurate before moving on to future steps. At this point, you can specify the delimiter, how many lines to skip at the start of the file, and whether there is no header row.When you are satisfied that the file is being processed correctly, click the Looks good, next button to move on to the next step.
In this step you are able to specify some more information about the structure of your data, specifically if this is time-series data or not.If it is time-series data, you will be presented with further configuration options where you can specify how the samples are arranged, the timestamp column, frequency of the timestamps, etc. This is important for correctly interpreting the data and ensuring that it is processed correctly in future steps.Click the Great, let's look at your values button to move on to the next step.
In this step, you can specify what information is contained in your CSV file, specifically which columns contain your values, labels, and metadata.If your CSV file contains metadata, and multiple rows will be grouped into a single generated sample in the following step, the metadata value for a given key (the column header) must stay consistent across the grouped rows. If they conflict, the import will fail so that the generated sample does not end up with incorrect metadata.Once you have completed your selections, click the Next, split up into samples button to move on to the next step.
In this section, you can set a length limit for generating samples when uploading data.For example, if you upload a CSV file that contains 30 seconds of data, and had configured a sample length limit of 3,000 ms in this step, the uploaded data would be split to create 10 distinct data samples of 3 seconds.You must specify how to deal with the label for the split samples, to handle the situation that may arise where one sample split contains rows with different labels. You can choose to use the last label in the sample, or to allow multiple labels for a single sample. For more information about multiple labels for a single sample, see our Multi-label documentation.To finalize your configuration, click the Finish wizard button.
Congratulations! You have successfully created a configuration file with the CSV Wizard. 🚀Data files that you upload into your project in future - whether it’s through the uploader, the CLI, the API or through data sources - will now be processed according to the configuration you set up with the CSV Wizard.
When you return to the CSV Wizard after creating a configuration, you will see a note at the top of the page indicating that you have an active CSV configuration, along with download links for the CSV file the configuration was created with and the JSON configuration file itself.
These files can also be downloaded from your project dashboard under the “Download block output” section. This is useful if you want to reuse a configuration across multiple projects or share it with other users.
To change your CSV Wizard configuration, simply go back to the CSV Wizard, upload a new reference file and repeat the Wizard steps again or upload an existing JSON configuration file. This will overwrite your current configuration with the new one.
When exporting CSV files from Azure services, you may need to limit your output by selecting a smaller set of subscriptions or time ranges.Azure Resource Graph has a limitation of 25,000 rows for CSV exports. Learn more by reading the Working with large Azure resource data sets documentation.
Large CSV files
If you’re working with large CSV files (over 25,000 rows), you may need to split them into smaller batches before uploading.For Windows users, you can use batch scripts or GUI tools to split large CSV files:
