LogoLogo
HomeDocsAPI & SDKsProjectsForumStudio
  • Welcome
    • Featured Machine Learning Projects
      • Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1
      • Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano
      • ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D
      • Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments
      • Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator
      • Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator
      • NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects
      • Community Guide – Using Edge Impulse with Nvidia DeepStream
      • Computer Vision Object Counting - Avnet RZBoard V2L
      • Gesture Appliances Control with Pose Detection - BrainChip AKD1000
      • Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)
      • High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)
    • Prototype and Concept Projects
      • Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning
      • TI CC1352P Launchpad - Getting Started with Machine Learning
      • OpenMV Cam RT1062 - Getting Started with Machine Learning
      • Getting Started with Edge Impulse Experiments
  • Computer Vision Projects
    • Workplace Organizer - Nvidia Jetson Nano
    • Recyclable Materials Sorter - Nvidia Jetson Nano
    • Analog Meter Reading - Arduino Nicla Vision
    • Creating Synthetic Data with Nvidia Omniverse Replicator
    • SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display
    • Traffic Monitoring - Brainchip Akida
    • Multi-camera Video Stream Inference - Brainchip Akida
    • Industrial Inspection Line - Brainchip Akida
    • X-Ray Classification and Analysis - Brainchip Akida
    • Inventory Stock Tracker - FOMO - BrainChip Akida
    • Container Counting - Arduino Nicla Vision
    • Smart Smoke Alarm - Arduino Nano 33
    • Shield Bot Autonomous Security Robot
    • Cyclist Blind Spot Detection - Himax WE-I Plus
    • IV Drip Fluid-Level Monitoring - Arduino Portenta H7
    • Worker PPE Safety Monitoring - Nvidia Jetson Nano
    • Delivered Package Detection - ESP-EYE
    • Bean Leaf Disease Classification - Sony Spresense
    • Oil Tank Measurement Using Computer Vision - Sony Spresense
    • Object Counting for Smart Industries - Raspberry Pi
    • Smart Cashier with FOMO - Raspberry Pi
    • PCB Defect Detection with Computer Vision - Raspberry Pi
    • Bicycle Counting - Sony Spresense
    • Counting Eggs with Computer Vision - OpenMV Cam H7
    • Elevator Passenger Counting - Arduino Nicla Vision
    • ESD Protection using Computer Vision - Seeed ReComputer
    • Solar Panel Defect Detection - Arduino Portenta H7
    • Label Defect Detection - Raspberry Pi
    • Dials and Knob Monitoring with Computer Vision - Raspberry Pi
    • Digital Character Recognition on Electric Meter System - OpenMV Cam H7
    • Corrosion Detection with Computer Vision - Seeed reTerminal
    • Inventory Management with Computer Vision - Raspberry Pi
    • Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L
    • Counting Retail Inventory with Computer Vision - Renesas RZ/V2L
    • Pose Detection - Renesas RZ/V2L
    • Product Quality Inspection - Renesas RZ/V2L
    • Smart Grocery Cart Using Computer Vision - OpenMV Cam H7
    • Driver Drowsiness Detection With FOMO - Arduino Nicla Vision
    • Gastroscopic Image Processing - OpenMV Cam H7
    • Pharmaceutical Pill Quality Control and Defect Detection
    • Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM
    • Smart Factory Prototype - Texas Instruments TDA4VM
    • Correct Posture Detection and Enforcement - Texas Instruments TDA4VM
    • Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM
    • Surface Crack Detection and Localization - Texas Instruments TDA4VM
    • Surface Crack Detection - Seeed reTerminal
    • Retail Image Classification - Nvidia Jetson Nano
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2
    • Object Detection and Visualization - Seeed Grove Vision AI Module
    • Bike Rearview Radar - Raspberry Pi
    • Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision
    • "Bring Your Own Model" Image Classifier for Wound Identification
    • Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano
    • Hardhat Detection in Industrial Settings - Alif Ensemble E7
    • Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A
    • Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A
    • Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense
    • Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5
    • Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit
    • Visual Anomaly Detection - Seeed Grove Vision AI Module V2
    • Object Counting with FOMO - OpenMV Cam RT1062
    • Visitor Heatmap with FOMO Object Detection - Jetson Orin Nano
    • Vehicle Security Camera - Arduino Portenta H7
  • Audio Projects
    • Occupancy Sensing - SiLabs xG24
    • Smart Appliance Control Using Voice Commands - Nordic Thingy:53
    • Glass Window Break Detection - Nordic Thingy:53
    • Illegal Logging Detection - Nordic Thingy:53
    • Illegal Logging Detection - Syntiant TinyML
    • Wearable Cough Sensor and Monitoring - Arduino Nano 33 BLE Sense
    • Collect Data for Keyword Spotting - Raspberry Pi Pico
    • Voice-Activated LED Strip - Raspberry Pi Pico
    • Snoring Detection on a Smart Phone
    • Gunshot Audio Classification - Arduino Nano 33 + Portenta H7
    • AI-Powered Patient Assistance - Arduino Nano 33 BLE Sense
    • Acoustic Pipe Leakage Detection - Arduino Portenta H7
    • Location Identification using Sound - Syntiant TinyML
    • Environmental Noise Classification - Nordic Thingy:53
    • Running Faucet Detection - Seeed XIAO Sense + Blues Cellular
    • Vandalism Detection via Audio Classification - Arduino Nano 33 BLE Sense
    • Predictive Maintenance Using Audio Classification - Arduino Nano 33 BLE Sense
    • Porting an Audio Project from the SiLabs Thunderboard Sense 2 to xG24
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 1
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 2
    • Keyword Spotting - Nordic Thingy:53
    • Detecting Worker Accidents with Audio Classification - Syntiant TinyML
    • Snoring Detection with Syntiant NDP120 Neural Decision Processor - Arduino Nicla Voice
    • Recognize Voice Commands with the Particle Photon 2
    • Voice Controlled Power Plug with Syntiant NDP120 (Nicla Voice)
    • Determining Compressor State with Audio Classification - Avnet RaSynBoard
    • Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline
    • Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier
  • Predictive Maintenance and Defect Detection Projects
    • Predictive Maintenance - Nordic Thingy:91
    • Brushless DC Motor Anomaly Detection
    • Industrial Compressor Predictive Maintenance - Nordic Thingy:53
    • Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense
    • Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal
    • Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal
    • Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal
    • Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549
    • BrickML Demo Project - 3D Printer Anomaly Detection
    • Condition Monitoring - Syntiant TinyML Board
    • Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense
    • Vibration Classification with BrainChip's Akida
    • AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32
  • Accelerometer and Activity Projects
    • Arduino x K-Way - Outdoor Activity Tracker
    • Arduino x K-Way - Gesture Recognition for Hiking
    • Arduino x K-Way - TinyML Fall Detection
    • Posture Detection for Worker Safety - SiLabs Thunderboard Sense 2
    • Hand Gesture Recognition - OpenMV Cam H7
    • Arduin-Row, a TinyML Rowing Machine Coach - Arduino Nicla Sense ME
    • Fall Detection using a Transformer Model – Arduino Giga R1 WiFi
    • Bluetooth Fall Detection - Arduino Nano 33 BLE Sense
    • Monitor Packages During Transit with AI - Arduino Nano 33 BLE Sense
    • Smart Baby Swing - Arduino Portenta H7
    • Warehouse Shipment Monitoring - SiLabs Thunderboard Sense 2
    • Gesture Recognition - Bangle.js Smartwatch
    • Gesture Recognition for Patient Communication - SiLabs Thunderboard Sense 2
    • Hospital Bed Occupancy Detection - Arduino Nano 33 BLE Sense
    • Porting a Posture Detection Project from the SiLabs Thunderboard Sense 2 to xG24
    • Porting a Gesture Recognition Project from the SiLabs Thunderboard Sense 2 to xG24
    • Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53
    • Classifying Exercise Activities on a BangleJS Smartwatch
  • Air Quality and Environmental Projects
    • Arduino x K-Way - Environmental Asthma Risk Assessment
    • Gas Detection in the Oil and Gas Industry - Nordic Thingy:91
    • Smart HVAC System with a Sony Spresense
    • Smart HVAC System with an Arduino Nicla Vision
    • Indoor CO2 Level Estimation - Arduino Portenta H7
    • Harmful Gases Detection - Arduino Nano 33 BLE Sense
    • Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense
    • AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3
    • AI-Assisted Air Quality Monitoring - DFRobot Firebeetle ESP32
    • Air Quality Monitoring with Sipeed Longan Nano - RISC-V Gigadevice
    • Methane Monitoring in Mines - Silabs xG24 Dev Kit
    • Smart Building Ventilation with Environmental Sensor Fusion
    • Sensor Data Fusion with Spresense and CommonSense
    • Water Pollution Detection - Arduino Nano ESP32 + Ultrasonic Scan
    • Fire Detection Using Sensor Fusion - Arduino Nano 33 BLE Sense
  • Novel Sensor Projects
    • 8x8 ToF Gesture Classification - Arduino RP2040 Connect
    • Food Irradiation Dose Detection - DFRobot Beetle ESP32C3
    • Applying EEG Data to Machine Learning, Part 1
    • Applying EEG Data to Machine Learning, Part 2
    • Applying EEG Data to Machine Learning, Part 3
    • Liquid Classification with TinyML - Seeed Wio Terminal + TDS Sensor
    • AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar
    • Soil Quality Detection Using AI and LoRaWAN - Seeed Sensecap A1101
    • Smart Diaper Prototype - Arduino Nicla Sense ME
    • DIY Smart Glove with Flex Sensors
    • EdgeML Energy Monitoring - Particle Photon 2
    • Wearable for Monitoring Worker Stress using HR/HRV DSP Block - Arduino Portenta
  • Software Integration Demos
    • Azure Machine Learning with Kubernetes Compute and Edge Impulse
    • ROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python
    • ROS2 + Edge Impulse, Part 2: MicroROS
    • Using Hugging Face Datasets in Edge Impulse
    • Using Hugging Face Image Classification Datasets with Edge Impulse
    • Edge Impulse API Usage Sample Application - Jetson Nano Trainer
    • MLOps with Edge Impulse and Azure IoT Edge
    • A Federated Approach to Train and Deploy Machine Learning Models
    • DIY Model Weight Update for Continuous AI Deployments
    • Automate the CI/CD Pipeline of your Models with Edge Impulse and GitHub Actions
    • Deploying Edge Impulse Models on ZEDEDA Cloud Devices
Powered by GitBook
On this page
  • Intro
  • Cloud ML with Azure ML
  • The Model
  • Getting Started with Azure ML
  • Jupyter Notebooks
  • ML Endpoints
  • Kubernetes Compute
  • Azure ML CLI & SDK
  • Edge ML with Edge Impulse
  • Hardware
  • Audio Project with Raspberry Pi
  • Data Collection
  • Training a Keyword Spotting Model
  • Raspberry Pi Deployment
  • Cloud ML Integration
  • Conclusions
  • References

Was this helpful?

Edit on GitHub
Export as PDF
  1. Software Integration Demos

Azure Machine Learning with Kubernetes Compute and Edge Impulse

Azure Machine Learning with Kubernetes Compute combined with Edge Impulse, for a sample device-to-cloud ML application.

PreviousSoftware Integration DemosNextROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python

Last updated 1 year ago

Was this helpful?

Created By: Attila Tokes

GitHub Repo:

Intro

Edge ML enables developers to run Machine Learning (ML) on Internet of Things (IoT) and Edge devices. This offers many advantages, such as reduced power consumption, low latency, reduced bandwidth, and increased privacy.

On the other hand, Edge ML can also be limited in functionality, given the reduced hardware capability of the edge devices. In these cases, it can be a good idea to combine Edge ML with Cloud ML functionality. This is usually done by running an ML model on the Edge device continuously, combined with a Cloud endpoint which is only called by the edge device when advanced functionality is needed.

In this project, I will demonstrate how to create a solution using Edge ML functionality provided by Edge Impulse, in combination with a Cloud ML endpoint implemented with Azure ML.

In this project we will implement a Voice-to-Text solution running on a low power edge device like the Raspberry Pi.

The device will be able to detect a keyword like "Listen!" and then record and translate voice to written text. The keyword detection will be implemented locally using an Edge Impulse model, while the voice-to-text transformation will use a model running in an Azure ML endpoint.

Below is short video showing the project in action:

In the following sections I will describe how such an application can be implemented. We will start with the voice-to-text endpoint implemented with Azure ML, and then we will integrate this into an Edge Impulse application running on the Raspberry Pi.

Cloud ML with Azure ML

In this section I will show how to implement a voice-to-text translation endpoint with Azure ML.

The Model

Wav2vec 2.0: Learning the structure of speech from raw audio https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/

Getting Started with Azure ML

Here we can create a new Workspace, if we don't already have one:

When we enter the workspace we want to use, a page with an overview and quick actions is shown:

Jupyter Notebooks

The compute instance can be created on-the-fly when we try to run the notebook:

(note: choosing the smallest and cheapest options should be sufficient)

It takes a couple of seconds for the instance to be started, after which we should be able to run the demo. What it does is:

  • downloads a sample audio file (WAV), with a person saying: "She had your duck soup and greasy washwater all year"

  • downloads a pre-trained version of the Wave2Vec 2.0 model (wav2vec2-base-960h)

  • runs the model on the sample audio file, and shows us the resulting transcript

ML Endpoints

Notebooks are a good way for experimenting with ML models. But, in order to make use of the functionality offered by the model, we need a way to expose the model for consumption by other components.

# The init() is called once at start-up
def init():
    # ... initialize model ...
   
# The run() method is called each time a request is made to the scoring API.
@input_schema(...)
@output_schema(...)
def run(data):    
    # ... run inference, and return the result ...

Using this file Azure ML will create a simple web server that exposes a /score endpoint. This endpoint can be accessed using a simple HTTP call.

The scoring file for our voice-to-text application can be found in the scoring-func/score_audio.py file.

  • first we need to select the "Custom" model type, and upload the scoring-func folder

  • then we choose a name

  • and register the model

Next we need an Environment in which the model can run. The Environment will define aspects such as the OS, Python version, and libraries installed in the Docker container where the Model will run.

There are two types of models we can use:

  • Curated Environments - these are ready-to-use environments created by Microsoft, and they have popular ML frameworks like TensorFlow or PyTorch pre-installed

  • Custom Environments - can be used we need custom libraries, or something that is not already present in the curated environments

This is PyTorch based image, supporting only CPU inference. It also has Python 3.7 and the transformers library installed.

  • choose a name, the compute type "Managed", and "Key-based authentication"

  • select the model we created earlier

  • on the Environment page we select our Custom Environment

  • choose a VM type, and set the Instance count to 1

  • review and confirm the settings, and click Create

The provisioning of our endpoint will take a couple of minutes. When the endpoint is ready it should like something like:

In order to consume the endpoint, we need to take note of the "REST endpoint" listed above, and as well the API Key from the Consume page:

Using these two pieces, we should now be able to make HTTP calls to our ML endpoint:

POST /score HTTP/1.1
Host: <name>.<region>.inference.ml.azure.com
Authorization: Bearer <api-key>
...
<data>

The endpoint accepts audio input as a numeric array in the data field. To call it with a real audio file we can use a client side Python script like this:

from scipy.io import wavfile
import requests
import librosa

# Read audio file
file_name = 'sample.wav'
input_audio, _ = librosa.load(file_name, sr=16000)

# Make request to the Azure ML endpoint
r = requests.post('https://ml-endpoint-voice-to-text.westeurope.inference.ml.azure.com/score',
             json={"data": input_audio.tolist()}, headers={"Authorization":"Bearer 4h6ic..."})

print("Status code: %d" % r.status_code)
print("Result: %s" % r.json())

This should produce the same result as the one seen in the Jupyter notebook example:

$ python3 sample-client.py

Status code: 200
Result: {'result': 'SHE HAD YOUR DUCK SOUP AND GREASY WASHWATER ALL YEAR'}

In the Monitoring tab we can see metrics like Request count and latency:

Kubernetes Compute

Up to this point, we have used the Managed Compute Instances / Clusters with Azure ML. Managed Compute Instances are Azure VM instances with lifecycle, OS updates, and software stacks fully managed by Azure. When using Managed Compute Instances we are able to select the VM instance type and size. Clusters can have either a fixed number of VM instances, or varying number of VM instances managed by auto-scaling functionality. Virtual Machines with dedicated GPUs are also supported.

Azure ML supports two types of Kubernetes compute clusters:

Running machine learning workloads on an already existing Kubernetes cluster can have many advantages, such as better resource utilization and scalability. On the other hand, setting up a Kubernetes compute cluster is not as easy, so using a managed solution can be helpful.

To set a Kubernetes compute in Azure ML, first we need to install the Azure Kubernetes ML Extension to our K8S cluster. For this project, I used a Azure Kubernetes Service (AKS cluster) which looked like this:

The Azure ML Extension can be installed using Azure CLI, by running the following command:

$ az k8s-extension create --name <cluster-name> --extension-type Microsoft.AzureML.Kubernetes --config enableTraining=True enableInference=True inferenceRouterServiceType=LoadBalancer allowInsecureConnections=True inferenceLoadBalancerHA=False --cluster-type managedClusters --cluster-name <cluster-name> --resource-group <resource-group> --scope cluster

Azure ML CLI & SDK

Azure ML Studio offers a good visual UI for creating and managing Azure ML resources. For people using Azure ML for the first time, it offers a great overview of how to get started and what features are available on the platform.

Additionally, Azure ML also has a CLI, and Python SDK for direct interaction from a console and code:

What is Azure Machine Learning CLI & Python SDK v2? https://docs.microsoft.com/en-us/azure/machine-learning/concept-v2

  • deployment automation

  • consistent and repeatable deployment

  • ability to create / manage / deploy resources programmatically

  • continuous integration and development (CI/CD)

Edge ML with Edge Impulse

As a demonstration we will implement a voice-to-text application on a Raspberry Pi. The solution will feature a keyword spotting model implemented with Edge Impulse, as as well the Cloud ML endpoint we created in the previous section.

Hardware

The hardware we will use is a Raspberry Pi 4 (2GB) development board, along with a Logitech USB headset used as the microphone input.

The Raspberry Pi 4 can be set up the standard way. The Raspberry Pi OS is flashed to an SD Card, then we set up network connectivity / Wifi and SSH access. The official documentation describes in great details how to do this:

Setting up your Raspberry Pi https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/0

Next, there are couple steps to be done in order to connect the device to EdgeImpulse. The goal is to install the edge-impulse-linux utility, which can be done as follows:

$ curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
$ sudo apt install -y gcc g++ make build-essential nodejs sox gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps
$ npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm

After running these commands, we should be able to connect to Edge Impulse Studio by running:

$ edge-impulse-linux --disable-camera

Audio Project with Raspberry Pi

and select the Audio project template

At this point, among other things, Studio offers us to connect a device to this project.

After we select "Connect your development board", we need to launch edge-impulse-linux on the Raspberry Pi:

The tool asks us to login to Edge Impulse, select a project and a microphone to be used. After completing these steps the device should show up the in the Devices tab:

Data Collection

Now we can start building our keyword spotting model. Edge Impulse has a great tutorial on this:

Responding to your voice https://docs.edgeimpulse.com/docs/tutorials/responding-to-your-voice

The first step in training a keyword spotting model is to collect a set of samples of the word we want to detect. This can be done in the Data Acquisition tab:

In our case the word we want to detect is "Listen!", so I collected about 3 minutes of audio data, which contained about ~ 130 samples of the word "Listen!":

Initially the data collection produces a single sample. This needs to be split up, so that each sample contains one instance of the word. Fortunately, this is easily done by selecting the Split Sample option from the context menu:

As a note, I ended up re-doing the data acquisition process, as I realized the recorded audio had a 50Hz mains interference noise picked up from the power supply of the Raspberry Pi. To fix this, I switched to using a power bank instead of a wall power supply and re-did the data collection.

To use these samples we can:

  • download the dataset to the Raspberry Pi

$ wget https://cdn.edgeimpulse.com/datasets/keywords2.zip
$ unzip keywords2.zip
  • reduce the number of samples to about ~130 per class (so that it matches the Listen! samples we have):

$ find noise/ | sort -R | tail +130 | xargs -n 1 -I % rm %
$ find unknown/ | sort -R | tail +130 | xargs -n 1 -I % rm %
$ edge-impulse-uploader --label noise --category split noise/*.wav
$ edge-impulse-uploader --label unknown --category split unknown/*.wav

The samples should appear in Edge Impulse, and we should see that samples for the 3 classes (listen, noise, unknown) are evenly distributed:

Training a Keyword Spotting Model

At this point our dataset is complete, and we can start building and training an ML pipeline / Impulse. This is relatively easy, as we can create an Impulse containing:

  • an Output block for our 3 classes

The overall accuracy I got is 96.8%, which is pretty good. In the Data Explorer section we can see that sample of our keyword (listen) are clearly separated from the unknown and noise samples.

Our Impulse at this point is ready to be used. We can try it out in the Live classification tab.

Raspberry Pi Deployment

The next step is to deploy the model as a standalone app on the Raspberry Pi. One way to do this is to use the edge-impulse-linux-runner app.

The edge-impulse-linux-runner tool automatically downloads and optimizes the model for the Raspberry Pi. Then it runs a sample app that continuously analyses the input audio, and gives the probabilities of the predicted classes:

$ sudo apt-get install libatlas-base-dev libportaudio2 libportaudiocpp0 portaudio19-dev
$ pip3 install edge_impulse_linux -i https://pypi.python.org/simple
$ pip3 install pyaudio

We can also get a set of examples by downloading the following GitHub repository:

$ git clone https://github.com/edgeimpulse/linux-sdk-python

An audio classification example app can found in the examples/audio/classify.py file. We can launch it as follows:

$ python3 linux-sdk-python/examples/audio/classify.py  /home/pi/.ei-linux-runner/models/128115/v1/model.eim

Cloud ML Integration

Now that we have the keyword spotting working, we can develop an app that also takes advantage of the Cloud ML functionality. So, using the Python SDK I created a simple app that does the following:

  • detects the "Listen!" keyword using the Edge Impulse model

  • when the keyword is spotted, records a couple seconds of audio

  • sends the recorded audio to the Cloud ML endpoint for voice-to-text transformation

  • displays the result / decoded text

This is what the output of the app looks like:

The app is built up from the following Python classes / files:

  • EdgeML / edgeml.py - responsible for running the keyword spotting model, until a given keyword is detected

  • Audio / audio.py - contains the audio recording functionality, with silence detection

  • CloudML / cloudml.py - responsible for talking to the Cloud ML endpoint

  • main.py - the entry point of the app, with a control loop linking the above parts together

The source code of the app can be found in the edgeml/python-app/ folder.

Conclusions

Using a combination of Edge ML and Cloud ML enables the creation of smart solutions with advanced functionality on low power edge devices. Edge ML is great for simpler tasks such as audio and signal processing, while Cloud ML enables the addition of more advanced functionality that would not otherwise be possible on edge devices.

Platforms like Edge Impulse and Azure ML enable developers to create machine learning solutions, without the need for deep knowledge of machine learning architectures and frameworks.

References

  1. Azure Machine Learning Documentation: https://docs.microsoft.com/en-us/azure/machine-learning/

  2. Edge Impulse Documentation: https://docs.edgeimpulse.com/docs/

  3. Wav2vec 2.0: Learning the structure of speech from raw audio: https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/

  4. Realizing Machine Learning anywhere with Azure Kubernetes Service and Arc-enabled Machine Learning: https://techcommunity.microsoft.com/t5/azure-arc-blog/realizing-machine-learning-anywhere-with-azure-kubernetes/ba-p/3470783

is Microsoft's cloud offering of machine learning services, covering the machine learning project lifecycle, including training and inference. It supports all the popular open-source machine learning frameworks such as TensorFlow, PyTorch and others.

The machine learning model we will use for voice-to-text transformation is the NLP framework, more specifically a pre-trained version of it. Wav2vec 2.0 is the second version of a speech recognition model developed by Facebook / Meta researchers:

A of Wav2vec 2.0 is available through the library. The pre-trained model supports both PyTorch and TensorFlow libraries. We will use it with PyTorch.

The functionality offered by Azure Machine Learning is accessed via the .

As a prerequisite to accessing Azure ML Studio, we will need an Azure account and an active Subscription. Users whoa are new to Azure can also create a , with one year of free services and some credits for experimentation.

Opening the brings us to a welcome page:

On the workspace overview page there are a couple of quick actions to choose from. I think Notebooks can be a good starting point. Notebooks allows us to work with a custom version of , a tool which should be familiar for most people involved with ML projects.

On the , we can either choose to create a new notebook, or to an upload existing one. I went ahead and created a new notebook, as I wanted to experiment with the Wave2Vec 2.0 model.

The "Wave2vec 2.0 Demo" notebook I used can be found .

The Notebooks interface is similar to that of a standard Jupyter install, but to run code we need an :

One way to do this is by using . Endpoints allows us to expose ML functionality over HTTPS endpoints, with features like SSL termination, authentication, DNS names and canary releases provided out-of-the-box.

In order to deploy an ML Endpoint we need to setup two things: a and an .

The Model contains a machine learning model packaged in some form. Supported formats are , and . The Score Model is the easiest option to implement. All we need is a Python "scoring file" of the following form:

We can upload this to Azure ML from the page:

As our model uses custom Python libraries like transformers, we need a Custom Environment. This can be created from the page. We can choose to start from a Curated Environment, or we can use our own Dockerfile. After multiple tries, I ended up creating a Custom Environment based on the mcr.microsoft.com/azureml/pytorch-1.10-ubuntu18.04-py37-cpu-inference image.

After this we should be ready to create an Endpoint. In the page we need to do the following:

Along with Managed Compute Instances, Azure ML also supports several other instance types. The most notable is based compute clusters. Kubernetes is a widely used open-source container orchestration system. It supports automatic deployment, scaling and management of container based application. Thus, it is a great choice for cloud-based systems.

cluster - these are fully managed clusters offered by Azure

enabled Kubernetes clusters - these are customer managed clusters connected to Azure via Arc

After the extension is installed successfully, the Kubernetes clusters can be attached to Azure ML from the view:

The attached Kubernetes Compute then can be used to create Endpoints with Kubernetes compute type:

The deployed ML Endpoint will look and work similar to one with managed compute type:

Using a kubectl CLI tool we can also see what resource Azure ML deployed in our Kubernetes Cluster:

The Azure ML CLI and Python SDK enable engineers the use techniques. Similar to DevOps, MLOps is a set of practices that allows the reliable and efficient management of AI / ML application lifecycle. It enables processes like:

is the leading development platform for Edge Machine Learning (Edge ML). It enables the creation of smart solutions via efficient machine learning models running on edge devices.

The Raspberry Pi 4 is a relatively low power single board computer, popular among makers. It is a . As a note, we are using a Raspberry Pi 4 mostly for convenience. The project probably could be implemented on any of the supported development boards with a microphone and Internet connectivity. The tools / programming languages may differ.

The full set of instructions can be found in the .

Next, we can login to the , and create a new project:

Along with the recorded keyword samples, we will also need some sample for other categories such as "Noise" and "Unknown" words. Luckily Edge Impulse already has a pre-built , which contains samples for these classes.

use the tool to upload the samples to our project:

a input with windows size of 1 sec

an processing block, which extracts cepstral coefficients from the audio data

a neural network based learning block

The block extracts coefficients from an audio signal. For keyword spotting, training it with the default parameters usually works:

The block is a neural network classifier, that takes the cepstral coefficients produced by the MFCC block, and tries to predict our 3 classes from it. We can train it with the default setting, but we also have the possibility to add some noise and randomness to the inputs:

If we want to modify / extend this application we can make use of the offered for Linux development boards. I opted for the Python SDK, which can be installed on the Raspberry Pi as follows:

Azure Machine Learning
Wav2vec 2.0
pre-trained version
🤗 Transformers
Azure ML Studio
free account
Azure ML Studio
Jupyter Notebook
Notebooks page
here
Azure Compute Instance
Azure Machine Learning Endpoints
Model
Environment
Score Model
MLFlow
Triton
Models
Environments
Endpoints
Kubernetes
Azure Kubernetes Service (AKS)
Azure Arc
MLOps
Edge Impulse
fully supported Edge Impulse development board
official guide
Edge Impulse Studio
keyword spotting dataset
Edge Impulse Uploader
Time series data
Audio MFCC
Classification (Keras)
MFCC _(Mel Frequency Cepstral Coefficients)_
NN Classifier
Edge Impulse SDKs
https://github.com/attila-tokes/edge-impulse-expert-projects/tree/main/azure-ml-voice-to-text
Compute