> For the complete documentation index, see [llms.txt](https://dev.solid-run.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://dev.solid-run.com/renesas/rz-v2n/rz-v2n-other-articles/rz-v2n-ai-autonomous-car-demo.md).
# RZ/V2N AI Autonomous Car Demo
### ๐ Overview
The **AI Dome โ RZ/V2N DRP-AI Autonomous Car** is a hands-on reference design that demonstrates how real-time computer vision and on-device AI inference can drive an autonomous vehicle without any cloud dependency. The demo combines object detection, visual tracking, and closed-loop motor control to search for a bowling pin, lock onto it, and drive directly to the target.
The platform is built around SolidRun's **HummingBoard-IIoT (HB-IIoT)** carrier paired with the **RZ/V2N SOM**, extended with a four-wheel robot chassis, LiDAR, camera, and an Arduino-based motor controller.
This project is intended for developers, robotics engineers, and embedded AI practitioners who want a practical starting point for building edge-AI robotics applications on the RZ/V series platform.
***
### ๐ฏ Demo Objectives
The demo is designed to showcase several capabilities at once:
* **Real-time object detection** of a bowling pin using a neural network accelerated by DRP-AI.
* **Autonomous search behavior** โ the car rotates and scans its environment until the target is found.
* **Visual servoing** โ once detected, the car centers the target in its field of view and drives toward it.
* **Fully edge-based inference** โ no network, no cloud, no external GPU.
* **Low-power AI compute** โ DRP-AI delivers high inference throughput at a fraction of the power budget of a typical GPU.
***
### ๐ง Hardware Platform
The demo is built on the **HummingBoard-IIoT (HB-IIoT)** carrier with the **RZ/V2N SOM**, integrated into a four-wheel robot chassis with LiDAR, camera, and Arduino-based motor control.
Full assembly instructions, including mechanical drawings, connector pinouts, and step-by-step build photos, are maintained in the SolidRun developer here:
๐ [**V2N AI Demo: Autonomous Car on HB-IIoT โ Setup & Assembly**](https://dev.solid-run.com/renesas/rz-v2n/rz-v2n-other-articles/rz-v2n-ai-autonomous-car-demo/autonomous-car-on-hb-iiot-setup-and-assembly)
***
The software pipeline runs cooperatively on the RZ/V2N, with low-level motor control delegated to the Arduino:
1. **Camera Capture** โ frames are pulled from the C920 at 30 FPS.
2. **Pre-processing** โ frames are resized and normalized for the AI model.
3. **DRP-AI Inference** โ an object detection model (e.g. YOLOX-Nano) runs on the DRP-AI accelerator.
4. **Post-processing** โ bounding boxes are decoded, filtered to the bowling-pin class, and scored.
5. **LiDAR Fusion** *(optional)* โ RPLIDAR data is used for obstacle avoidance.
6. **Control Logic** โ a state machine decides whether to **search**, **align**, **drive**, or **stop**.
7. **Motor Commands** โ the RZ/V2N sends high-level commands over USB/serial to the Arduino, which translates them into PWM signals for the Motor Shield V5.3.
#### High-Level Data Flow
```
[C920 Camera] โโโบ [Pre-process] โโโบ [DRP-AI Inference] โโโบ [Post-process]
โ
[RPLIDAR] โโโโโโโโโโโโโโโโโโโโโโโโโโโค
โผ
[State Machine]
โ
โผ
[Serial: RZ/V2N โ Arduino Uno]
โ
โผ
[Motor Shield V5.3] โโโบ [4 ร DC Motors]
```
***
### ๐ Behavior State Machine
#### 1. SEARCH
The car rotates slowly in place while scanning every frame for a bowling pin.
#### 2. ALIGN
Once a pin is detected, the car computes the horizontal offset between the bounding-box center and the image center, then turns until the offset is within a small deadband.
#### 3. DRIVE
With the target centered, the car moves forward and continuously re-checks the target's position, re-entering ALIGN if drift occurs.
#### 4. STOP
When the bounding box grows beyond a target-size threshold (i.e. the pin is close), the car halts.
***
### โ๏ธ Software Setup
#### Arduino (Motor Controller)
The Arduino handles the low-level PWM and encoder feedback. The firmware source lives in the SolidRun GitHub repository.
**Prerequisites**
* USB cable (Arduino โ host PC or RZ/V2N EVK)
* `git`
* `python3`
* [PlatformIO CLI](https://platformio.org/install/cli)
**Flashing the Arduino Code**
bash
```bash
# 1. Clone the motor-controller repository
git clone https://github.com/SolidRun/rzv2n-arduino-motor-controller.git
cd rzv2n-arduino-motor-controller
# 2. Verify the project environment
pio project config
# Ensure a valid environment exists (e.g. `uno`)
# 3. Connect the Arduino UNO via USB and identify its port
pio device list
# Note the serial port (e.g. /dev/ttyACM0)
# 4. Build
pio run
# 5. Upload (flash)
pio run --target upload
```
#### RZ/V2N Application
The AI application runs on the RZ/V2N side and talks to the Arduino over USB/serial.
**Prerequisites**
* Yocto-based Linux BSP for RZ/V2N
* DRP-AI Translator / DRP-AI TVM toolchain
* Cross-compile toolchain (AArch64 GCC)
**Build & Deploy**
bash
```bash
git clone https://github.com/your-org/rzv2n-ai-dome-car.git
cd rzv2n-ai-dome-car
source /opt/poky//environment-setup-aarch64-poky-linux
make all
scp build/ai_dome_car root@:/home/root/
```
**Run the Demo**
On the target board:
bash
```bash
./ai_dome_car --model models/yolox_nano_pin.drpai \
--camera /dev/video0 \
--serial /dev/ttyACM0 \
--confidence 0.5
```
Place a bowling pin in front of the car and power the chassis. The car begins in SEARCH, locks onto the pin, and drives toward it.
***
### ๐งช Model Details
* **Architecture:** YOLOX-Nano (or equivalent lightweight detector)
* **Input Resolution:** 416 ร 416
* **Classes:** single-class (`bowling_pin`), fine-tuned from a COCO-pretrained backbone
* **Quantization:** INT8 via DRP-AI Translator
* **Inference Time:** \~15 ms per frame on DRP-AI
* **Accuracy:** >90% mAP on the custom bowling-pin dataset
Training notebooks and the custom dataset live in the `training/` directory of the repo.
***
### ๐ Calibration & Tuning
All tunable parameters live in `config/params.yaml`:
* `CONFIDENCE_THRESHOLD` โ minimum detection score to consider a valid target
* `ALIGN_DEADBAND_PX` โ horizontal pixel tolerance before switching ALIGN โ DRIVE
* `STOP_BBOX_HEIGHT` โ bounding-box height that triggers STOP
* `SEARCH_SPIN_SPEED` โ rotation speed during SEARCH
* `DRIVE_SPEED` โ forward speed during DRIVE
* `SERIAL_PORT` โ path to the Arduino serial device
### ๐ Troubleshooting
| Issue | Likely Cause | Suggested Fix |
| -------------------------------- | ----------------------------------- | --------------------------------------------------------- |
| No detections | Poor lighting or threshold too high | Improve lighting; lower `CONFIDENCE_THRESHOLD` |
| Car oscillates when aligning | Deadband too tight | Increase `ALIGN_DEADBAND_PX` |
| Car drives past the pin | Stop threshold too small | Increase `STOP_BBOX_HEIGHT` |
| Low frame rate | Model not running on DRP-AI | Confirm the `.drpai` binary is loaded (not ONNX fallback) |
| Motors jitter | PWM mismatch or loose encoder cable | Re-seat encoder connector; verify PWM config |
| Arduino not detected | Wrong serial port | Re-run `pio device list`; update `--serial` flag |
| Car moves in the wrong direction | Motor IN1/IN2 swapped | Verify wiring against the Motor Placement table |
***
### ๐ References
* [SolidRun โ V2N AI Demo: Autonomous Car on HB-IIoT โ Setup & Assembly](https://developer.resources.solid-run.com/wiki/spaces/developer/pages/1696890899/V2N+AI+Demo+Autonomous+Car+on+HB-IIoT+Setup+Assembly)
* [SolidRun โ HummingBoard IIoT & RZ/V2N SOM Quick Start Guide](https://developer.resources.solid-run.com/wiki/spaces/developer/pages/1526005761/HummingBoard+IIOT+RZ+V2N+SOM+Quick+Start+Guide)
* [SolidRun โ RZ/V2N SOM Hardware User Manual](https://developer.resources.solid-run.com/wiki/spaces/developer/pages/1709178894/RZ+V2N+SOM+Hardware+User+Manual)
* [SolidRun GitHub โ rzv2n-arduino-motor-controller](https://github.com/SolidRun/rzv2n-arduino-motor-controller)
* Renesas RZ/V2N product page and datasheet
* DRP-AI Translator and DRP-AI TVM documentation
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://dev.solid-run.com/renesas/rz-v2n/rz-v2n-other-articles/rz-v2n-ai-autonomous-car-demo.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.