Summarize the AI application object detection development process using Qualcomm AI Hub and Qualcomm Linux on RB3 Gen 2. The gst-ai-object-detection application allows you to detect objects within images and videos. The use cases show the execution of YOLOv8 using LiteRT.
Overview of the AI Use Case and the Selected AI Model to Deploy
- YOLOv8-Detection
- Quantized real‑time object detection optimized for mobile and edge by Ultralytics.
- Ultralytics YOLOv8 is a machine learning model that predicts bounding boxes and classes of objects in an image.
- This model is post‑training quantized to int8 using samples from the COCO dataset.
Steps to optimize, validate, and deploy selected AI models on hosted Qualcomm devices through Qualcomm AI Hub
- Hosted target device to verify: RB3 Gen 2
- Set up an environment for Qualcomm AI Hub on your host machine (Ubuntu22.04)
- AI Hub on-device deployment :
python -m qai_hub_models.models.yolov8_det.export --quantize w8a8 --device "RB3 Gen 2 (Proxy)"
- Run with:
python -m qai_hub_models.models.yolov8_det.export --quantize w8a8 --device "RB3 Gen 2 (Proxy)"
- Compile job result
-
Profile job result
-
Inference job result
Steps to deploy the AI model from Qualcomm AI Hub to a physical device
- Physical device to deploy : RB3 Gen 2 Vision kit
- OS : Qualcomm Linux 1.5
- Set up the host machine for Android Debug Bridge (ADB) connection to the RB3 Gen 2 Vision kit.
- Connect the RB3 Gen 2 Vision kit to a monitor through HDMI.
- Download and copy the models, labels file, and image to the /etc/ directory on the device
- Download AI model from Qualcomm AI Hub
- Visit previous profile job webpage
- Download TFlite Runtime model
-
Use the following command to enter the shell and execute the use cases:
adb shell -
Run the following command to enable the Permissive mode:
setenforce 0 -
Run the following command to enable the display:
export XDG_RUNTIME_DIR=/dev/socket/weston && export WAYLAND_DISPLAY=wayland-1 -
Download the labels file or you have your own labels file
-
Prepare your own test_video file
-
Run the following command to push the files from host machine:
adb push coco_labels.txt /etc adb push YOLOv8-Detection-Quantized.tflite /etc adb push test_video.mp4 /etc -
Or you can push your own model to /etc
-
The sample application uses the /etc/config_detection.json file to read the input parameters.
-
Create the /etc/config_detection.json file
{
"file-path": "<input video path>",
"ml-framework": "<snpe, or tflite or qnn framework>",
"yolo-model-type": "<yolov8 or yolonas or yolov5 or yolox>",
"model": "<Model Path>",
"labels": "<Label Path>",
"constants": "<Model Constants for LiteRT Model>",
"threshold": <Post-processing threshold, integer value from 1-100>,
"runtime": "<dsp, cpu or gpu runtime>"
}- For information on updating the q_offset and q_scale constants of the quantized model:
- Open the downloaded TFLite model with a graph viewer tool like Netron and open model. Check post-processing requirements for the specific model, which needs to be updated into reference application.
- Here is a screenshot of above-mentioned model. Click on input node of the model to see model properties. Copy the values from output node of the model.
- Use the following config(for Qualcomm linux 1.5) to execute the use case (camera source):
{
"camera": 0,
"ml-framework": "tflite",
"yolo-model-type": "yolov8",
"model": "/etc/YOLOv8-Detection-Quantized.tflite ",
"labels": "/etc/coco_labels.txt",
"constants": "YOLOv8,q-offsets=<21.0, 0.0, 0.0>,q-scales=<3.0546178817749023, 0.003793874057009816, 1.0>;",
"threshold": 40,
"runtime": "dsp"
}- Use the following config(for Qualcomm linux 1.5) to execute the use case (video source):
{
"file-path": "/etc/test_video.mp4",
"ml-framework": "tflite",
"yolo-model-type": "yolov8",
"model": "/etc/YOLOv8-Detection-Quantized.tflite ",
"labels": "/etc/coco_labels.txt",
"constants": "YOLOv8,q-offsets=<21.0, 0.0, 0.0>,q-scales=<3.0546178817749023, 0.003793874057009816, 1.0>;",
"threshold": 40,
"runtime": "dsp"
}-
Running object detection with QIMSDK and TFLite
gst-ai-object-detection --config-file=/etc/config_detection.json -
To stop the use case, press CTRL + C
-
To display the available help options, run the following command:
gst-ai-object-detection –h -
The GStreamer debug output is controlled by the GST_DEBUG environment variable. Set the required level to enable logging. For example, to log all warnings, run the following command.
export GST_DEBUG=2
For more detail: Qualcomm intelligent multimedia software development kit (QIM SDK) This SDK uses GStreamer, an open-source multimedia framework and exposes easy APIs and plug-ins in both multimedia and AI/ML domain.
