Generate audio data using the Eleven Labs block, specifically for non-human voice audio. In this guide and video we'll be exploring an innovative approach leveraging generative AI to train small edge AI models. This is particularly useful in scenarios where high-quality, diverse training data is scarce or expensive to collect.
There is also a video version of this guide:
Prerequisites
Eleven Labs: account and API Key
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
Transformation blocks can be complex to set up and are one of the most advanced features Edge Impulse provides. Feel free to ask your customer solution engineer for some help and some examples, we have been setting up complex pipelines for our customers and our engineers have acquired a lot of expertise with transformation blocks.
Introduction
Edge AI enables smart devices to perform machine learning tasks right at the source of the data collection. These small models are very good and optimized for performance scope tasks, as long as they are trained on quality data. We usually say "garbage in, garbage out."
Data Quality and Diversity
To help us improve the quality of our sound datasets, we've been working on an integration between Edge Impulse and ElevenLabs.io. Edge Impulse excels in the creation and optimization of Edge AI models, while Eleven Labs offers advanced capabilities to create realistic sound effects. This integration allows us to expand our datasets with sounds that may be difficult or expensive to record naturally. This approach not only saves time and money but also enhances the accuracy and reliability of the models we deploy on edge devices.
Practical Application: Glass Breaking Sound
In the above demonstration, we focus on a practical application that can be used in a smart security system, or in a factory to detect incidents, such as detecting the sounds of glass breaking.
Getting Started
Steps to Generate Sound Samples with ElevenLabs.io
Navigate to Data Acquisition: Once you're in your project, navigate to the Data Acquisition section and go to Data Sources. Under Data Sources, you can add a new one and select the transformation block for sound generation using generative AI.
Step 1: Generate a Sound Sample with ElevenLabs.io
First, get your Eleven Labs API Key. Navigate to the Eleven Labs web interface to get your key and test your prompt.
Step 2: Define the Prompt
Here we will be trying to collect a glass breaking sound or impact. Navigate to the settings to define the prompt "glass breaking" and define the length (e.g., 2 seconds) and prompt influence
Step 3: Run the Pipeline:
Once you've set up your prompt, and api key, run the pipeline to generate the sound samples. You can then view the output in the Data Acquisition section.
Benefits of Using Generative AI for Sound Generation
Enhance Data Quality: Generative AI can create high-quality sound samples that are difficult to record naturally.
Increase Dataset Diversity: Access a wide range of sounds to enrich your training dataset and improve model performance.
Save Time and Resources: Quickly generate the sound samples you need without the hassle of manual recording.
Improve Model Accuracy: High-quality, diverse sound samples can help fill gaps in your dataset and enhance model performance.
Conclusion
By leveraging generative AI for sound generation, you can enhance the quality and diversity of your training datasets, leading to more accurate and reliable edge AI models. This innovative approach saves time and resources while improving the performance of your models in real-world applications. Try out the Eleven Labs block in Edge Impulse today and start creating high-quality sound datasets for your projects.
Synthetic datasets are managed through our Synthetic Data tab. This tab provides a user-friendly interface to generate synthetic data for your projects. You can create synthetic datasets using a variety of tools and models, such as DALL-E for image generation or Whisper for human speech synthesis.
We have put together the following tutorials to help you get started with synthetic datasets generation:
Synthetic datasets are a collection of data artificially generated rather than being collected from real-world observations or measurements. They are created using algorithms, simulations, or mathematical models to mimic the characteristics and patterns of real data. Synthetic datasets are a valuable tool to generate data for experimentation, testing, and development when obtaining real data is challenging, costly, or undesirable.
You might want to generate synthetic datasets for several reasons:
Cost Efficiency: Creating synthetic data can be more cost-effective and efficient than collecting large volumes of real data, especially in resource-constrained environments.
Data Augmentation: Synthetic datasets allow users to augment their real-world data with variations, which can improve model robustness and performance.
Data Diversity: Synthetic datasets enable the inclusion of uncommon or rare scenarios, enriching model training with a wider range of potential inputs.
Privacy and Security: When dealing with sensitive data, synthetic datasets provide a way to train models without exposing real information, enhancing privacy and security.
The Synthetic Data tab allows you to easily create and manage synthetic data, enhancing your datasets and improving model performance. Whether you need images, speech, or audio data, our new integrations make it simple and efficient.
There is also a video version of this guide Placeholder video TBD:
DALL-E Image Generation Block: Generate image datasets using Dall·E using the DALL-E model.
Whisper Keyword Spotting Generation Block: Generate keyword-spotting datasets using the Whisper model. Ideal for keyword spotting and speech recognition applications.
To use these features, navigate to Data Sources, add new data source transformation blocks, set up actions, run a pipeline, and then go to Data Acquisition to view the output. If you want to make changes or refine your prompts, you have to delete the pipeline and start over.
Enhance Your Datasets: Easily augment your datasets with high-quality synthetic data.
Improve Model Accuracy: Synthetic data can help fill gaps in your dataset, leading to better model performance.
Save Time and Resources: Quickly generate the data you need without the hassle of manual data collection.
Data ingestion will also include a flag in the header "synthetic": true
, allowing users to pass an optional new header to indicate this is synthetic data. We've updated the Whisper and DALL-E blocks to accommodate this update.
To access the Synthetic Data tab, follow these steps:
Navigate to Your Project: Open your project in Edge Impulse Studio.
Open Synthetic Data Tab: Click on the "Synthetic Data" tab in the left-hand menu.
Create Realistic Images: Use DALL-E to generate realistic images for your datasets.
Customize Prompts: Tailor the prompts to generate specific types of images suited to your project needs.
Select Image Generation: Choose the GPT-4 (DALL-E) option.
Enter a Prompt: Describe the type of images you need (e.g., "A photo of a factory worker wearing a hard hat").
Generate and Save: Click "Generate" to create the images. Review and save the generated images to your dataset.
Human-like Speech Data: Utilize Whisper to generate human-like speech data.
Versatile Applications: Ideal for voice recognition, command-and-control systems, or any application requiring natural language processing.
Select Speech Generation: Choose the Whisper option.
Enter Text: Provide the text you want to be converted into speech (e.g., "Hello Edge!").
Generate and Save: Click "Generate" to create the speech data. Review and save the generated audio files.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
Transformation blocks can be complex to set up and are one of the most advanced features Edge Impulse provides. Feel free to ask your customer solution engineer for some help and some examples, we have been setting up complex pipelines for our customers and our engineers have acquired a lot of expertise with transformation blocks.
To start using the Synthetic Data tab, log in to your Edge Impulse Enterprise account and open a project. Navigate to the "Synthetic Data" tab and explore the new features. If you don't have an account yet, sign up for free at Edge Impulse.
For further assistance, visit our forum or check out our Introduction to Edge AI Course.
Stay tuned for more updates on what we're doing with generative AI. Exciting times ahead!
This notebook takes you through a basic example of using the physics simulation tool PyBullet to generate an accelerometer dataset representing dropping the Nordic Thingy:53 devkit from different heights. This dataset can be used to train a regression model to predict drop height.
This idea could be used for a wide range of simulatable environments- for example generating accelerometer datasets for pose estimation or fall detection. The same concept could be applied in an FMEA application for generating strain datasets for structural monitoring.
There is also a video version of this tutorial:
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
The dependencies can be installed with:
We need to load in a Universal Robotics Description Format file describing an object with the dimensions and weight of a Nordic Thingy:53. In this case, measuring our device it is 64x60x23.5mm and its weight 60g. The shape is given by a .obj 3D model file.
To generate the required data we will be running PyBullet in headless "DIRECT" mode so we can iterate quickly over the parameter field. If you run the python file below you can see how pybullet simulates the object dropping onto a plane
First off we need to set up a pybullet physics simulation environment. We load in our object file and a plane for it to drop onto. The plane's dynamics can be adjusted to better represent the real world (in this case we're dropping onto carpet)
We also need to define the output folder for our simulated accelerometer files
And define the drop parameters
We also need to define the characteristics of the IMU on the real device we are trying to simulate. In this case the Nordic Thingy:53 has a Bosch BMI270 IMU (https://www.bosch-sensortec.com/products/motion-sensors/imus/bmi270/) which is set to a range of +-2g with a resolution of 0.06g. These parameters will be used to restrict the raw acceleration output:
Finally we are going to give the object and plane restitution properties to allow for some bounce. In this case I dropped the real Thingy:53 onto a hardwood table. You can use p.changeDynamics to introduce other factors such as damping and friction.
Here we iterate over a range of heights, randomly changing its start orientation for i number of simulations per height. The acceleration is calculated relative to the orientation of the Thingy:53 object to represent its onboard accelerometer.
Finally we save the metadata file to the output folder. This can be used to tell the edge-impulse-uploader CLI tool the floating point labels for each file.
These files can then be uploaded to a project with these commands (run in a separate terminal window):
(run edge-impulse-uploader --clean if you have used the CLI before to reset the target project)
Now you can use your dataset a drop height detection regression model in Edge Impulse Studio!
See if you can edit this project to simulate throwing the object up in the air to predict the maximum height, or add in your own custom object. You could also try to better model the real environment you're dropping the object in- adding air resistance, friction, damping and material properties for your surface.
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
pip packages (install with pip install
packagename
):
pydub https://pypi.org/project/pydub/
google-cloud-texttospeech https://cloud.google.com/python/docs/reference/texttospeech/latest
requests https://pypi.org/project/requests/
First off you will need to set up and Edge Impulse account and create your first project. You will also need a Google Cloud account with the Text to Speech API enabled: https://cloud.google.com/text-to-speech, the first million characters generated each month are free (WaveNet voices), this should be plenty for most cases as you'll only need to generate your dataset once. From google you will need to download a credentials JSON file and set it to the correct environment variable on your system to allow the python API to work: (https://developers.google.com/workspace/guides/create-credentials#service-account)
First off we need to set our desired keywords and labels:
Then we need to set up the parameters for our speech dataset, all possible combinations will be iterated through:
languages - Choose the text to speech voice languages to use (https://cloud.google.com/text-to-speech/docs/voices)
pitches - Which voice pitches to apply
genders - Which SSML genders to apply
speakingRates - Which speaking speeds to apply
Then provide some other key parameters:
out_length - How long each output sample should be
count - Maximum number of samples to output (if all combinations of languages, pitches etc are higher then this restricts output)
voice-dir - Where to store the clean samples before noise is added
noise-url - Which noise file to download and apply to your samples
output-folder - The final output location of the noised samples
num-copies - How many different noisy versions of each sample to create
max-noise-level - in Db,
Then we need to check all the output folders are ready
And download the background noise file
Then we can generate a list of all possible parameter combinations based on the input earlier. If you have set num_copies
to be smaller than the number of combinations then these options will be reduced:
Finally we iterate though all the options generated, call the Google TTS API to generate the desired sample, and apply noise to it, saving locally with metadata:
The files in ./out-noisy
can be uploaded easily using the Edge Impulse CLI tool:
Now you can use your keywords to create a robust keyword detection model in Edge Impulse Studio!
Make use of our pre-built keyword dataset to add noise and 'unknown' words to your model: Keyword Spotting Dataset
Try out both classification models and the transfer learning keyword spotting model to see which works best for your case
This notebook explores how we can use generative AI to create datasets which don't exist yet. This can be a good starting point for your project if you have not collected or cannot collect the data required. It is important to note the limitations of generative AI still apply here, biases can be introduced through your prompts, results can include "hallucinations" and quality control is important.
This example uses the OpenAI API to call the Dall-E image generation tool, it explores both generation and variation but there are other tools such as editing which could also be useful for augmenting an existing dataset.
There is also a video version of this tutorial:
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
pip packages (install with pip install
packagename
):
openai https://pypi.org/project/openai/
First off you will need to set up and Edge Impulse account and create your first project.
You will also need to create an API Key for OpenAI: https://platform.openai.com/docs/api-reference/authentication
The API takes in a prompt, number of images and a size
The API also has a variations call which takes in an existing images and creates variations of it. This could also be used to modify existing images.
Here we are iterate through a number of images and variations to generate a dataset based on the prompts/labels given.
These files can then be uploaded to a project with these commands (run in a separate terminal window):
(run edge-impulse-uploader --clean if you have used the CLI before to reset the target project)
Now you can use your images to create an image classification model on Edge Impulse.
Why not try some other OpenAI calls, 'edit' could be used to take an existing image and translate it into different environments or add different humans to increase the variety of your dataset. https://platform.openai.com/docs/guides/images/usage
We have wrapped this example into a (Enterprise Feature) to make it even easier to generate images and upload them to your organization. See: