Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision

Convert an R/C Car to a self-driving vehicle that uses computer vision to identify and follow a series of target objects.

Created By: Jeremy Ellis

Public Project Link:

Introduction

As a teacher, I wanted a long-term project for students who were willing to do extra work and who showed an interest in both Arduino microcontrollers and Computer Vision / Machine Learning. We took an off-the-shelf RC car, and converted it to be controlled by an Arduino Portenta H7, and then trained an Edge Impulse FOMO Vision model to create a self-driving RoboCar that can follow a path of target objects. This high-level, long-term project is great for students who want a harder-than-normal challenge.

We picked a specific RC car, the "", as it did not use a brushless motor (the programming of brushless motors has extra complexity). We also needed a big motor driver, a servo motor (the Shredder steering motor is a DC motor with a potentiometer), connectors, USB cable and USB battery pack, as well as the ability to 3D Print large objects. See the BOM and price list at the bottom of this post.

This is a difficult build and may need some adaptations for your project: For example, we used a large print-bed 3D printer to make the 10.5 inch long frame (26.7 cm). You may choose to split that frame up or put the hardware somewhere else on the car.

Main Build Steps

1. Purchase Parts

First, you need to purchase the parts required to build the car. Check out the BOM provided below, and make sure to purchase all the required parts without exceeding your budget.

List of suggested first-time purchases:

USB-C cable, USB battery pack, microSD card. A 3D printer would be useful.

2. Test Vehicle

Have fun with the RC car: Without breaking it, charge the battery and drive it around to test the tight turning circle diameter and slowest speeds that it still moves.

3. Find Objects to Track

4. Build the Frame

5. Train the Model

On the dashboard, go to the far right and change the label type from "one label" to "bounding boxes." This sets up the model for FOMO (faster objects, more objects) object detection. Also, set Target Device to "Arduino Portenta H7 (Cortex M7 480 MHz)"

On the Dashboard, click on Add Existing Data" and upload your images that you took of the car driving around your object.
Go to Data Acquisition and click on the "Labeling Queue" and start drawing bounding boxes around your objects. Make sure the label is the same for all of them. I use "1eye". The number in front of your label is very important for an easy way to determine which label you are analyzing.

Once all of the images are labelled, create a FOMO model by clicking Create impulse and choose the Edge Impulse Image and Object Detection blocks. Double check before saving the Impulse that it has only one Object block.
On the next page, for the menu item "Image", make sure select Grayscale, not RGB, and save it.
Click on "Generate Features", wait a few minutes for the "Feature explorer" to be built, then check for any outliers and delete or change them as needed.
In the menu item "Object Detection" set the number of training cycles to about 200 (or whatever works for your project). Add a check in the box for "Data Augmentation" and click "Start Training". Depending on the final number of images, this will take about 8 minutes.
Once completed, check the F1 Score and Confusion matrix, which should hopefully be above 80% for the Background and the "1eye" labelled object.
Flash the model to the Portenta connected to your car. Make sure both batteries are charged. When testing directly on the vehicle, always securely place the car on a small box when testing, as it could jump off the table and break.
Always unplug both batteries when you have finished testing the car.

6. Add the Model to the Application

Download the Arduino library from the Edge Impulse Deployment page, and install the .zip file into your Arduino IDE (Sketch--> Include Library --> Add .ZIP Library). This could be a long compile depending on your computer's speed. Expect about 20 minutes for this to complete the first time, and about 3 minutes each time after that. Ensuring the code compiles and loading it onto the Portenta is a huge step. Well done if it works!

7. Load the Arduino Sketch

8. Waveshare OLED (Optional)

If using the optional (but very useful for debugging) WaveShare grayscale 128 x 128 OLED, then attach the correct pins to the Portenta:

black to GND,
red to 3v3,
blue DIN (mosi) to D8,
yellow (sck) to D9,
orange (cs) to D7,
green (dc) to D6,
white (reset) not needed

9. Vehicle Modifications

servo red to 6V battery positive
servo ground (brown or black) to 6V battery negative
servo ground (brown or black) to board GND
servo controller (orange or other color) to board PWM pin D2

Here is an image of the entire wiring diagram:

See image for pin connections:

Left side of Big Motor Driver board in order from the top (see diagram): D3 - D4 - - D5 3V3 GND - -

Make sure the car voltage is kept away from the 5-volt USB input for the Portenta. A tricky part of the car build is getting the main battery power to the big motor driver, and deciding if the main battery is going to power the Portenta. We tried step-down 5V regulators, but the Portenta seemed very sensitive to power fluctuations when the car was running so I just powered the Portenta with a 5V USB battery pack attached to the top of the breadboard. Your big motor driver will need connectors for both the main battery and the motor.

10. Vehicle Testing

Improvements and Contributions

As everything here is open-source, feel free to improve the code, submit pull requests, share on social media and tag me on Twitter @rocksetta!

Here are some of the student's cars built so far:

Conclusion

I hope you enjoyed this difficult project, it involved connecting an Arduino Portenta H7 to a DC motor and Servo motor, adding with Machine Learning using Edge Impulse, and problem-solving to improve the Computer Vision model. When you have the roboCar working, students can start making extensions, such as lighting and battery monitoring, including adaptions to help the car recover if it loses the trail and is unsure of the correct path to follow. Good luck and I hope you had fun!