香橙派rk3588在ubuntu上测试rknn的C++调用npu例程rknn_yolov5_video_demo

接上篇提到的视频推理例子rknn_yolov5_video_demo
rknn_yolov5_video_demo是和rknn_yolov5_demo一起编译出来的
编译过程看上篇 本篇只谈使用

rknn_yolov5_video_demo
这个程序会加载模型和读取输入视频 将视频解码成图片 对图片进行预处理 然后进行推理 根据推理结果对图片进行后处理 再输出带有推理信息的图片 最后把图片编码成视频
程序支持读取h264 h265 和拉流rtsp 其中拉流rtsp不能在安卓系统上使用
注意h264 h265和平时常见的mp4不一样 需要用ffmpeg处理一下

直接在3588上安装ffmpeg
sudo apt install ffmpeg

ffmpeg将mp4处理成h264
ffmpeg -i xxx.mp4 -vcodec h264 xxx.h264

在rk3588上处理比较慢 如果时间太长可以按q提前退出 这样可以只转换一小段时间的h264
也可以在pc上装ffmpeg转换 pc转换快
建议输入视频时间不要太长 不然后面程序处理视频也要很久

执行程序
./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn xxx.h264 264

输出信息
orangepi@orangepi5ultra:~/pro/rknn-toolkit2/rknpu2/examples/rknn_yolov5_demo/install/rknn_yolov5_demo_Linux$ ./rknn_yolov5_video_demo model/RK3588/yolov5s-640-640.rknn LasVegas.h264 264
Loading mode...
sdk version: 2.0.0b0 (35a6907d79@2024-03-24T10:31:14) driver version: 0.9.6
model input num: 1, output num: 3
index=0, name=images, n_dims=4, dims=[1, 640, 640, 3], n_elems=1228800, size=1228800, fmt=NHWC, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=0, name=output0, n_dims=4, dims=[1, 255, 80, 80], n_elems=1632000, size=1632000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=1, name=286, n_dims=4, dims=[1, 255, 40, 40], n_elems=408000, size=408000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
index=2, name=288, n_dims=4, dims=[1, 255, 20, 20], n_elems=102000, size=102000, fmt=NCHW, type=INT8, qnt_type=AFFINE, zp=-128, scale=0.003922
model is NHWC input fmt
model input height=640, width=640, channel=3
mpi_dec_test start mpi_dec_test decoder test start mpp_type 7 app_ctx=0x7fcf0c23b0 decoder=0x55c98100f0
read video size=51907411
receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 receive packet size=8192 decoder require buffer w:h [1920:1080] stride [1920:1088] buf_size 4177920 pts=0 dts=0 decode_get_frame get info changed found receive packet size=8192 receive packet size=8192 receive packet size=8192 decoder require buffer w:h [1920:1080] stride [1920:1088] buf_size 4177920 pts=0 dts=0 get one frame 1745683729681 data_vir=0x7fa511a000 fd=14 0x55c9d2fe20 encoder test start w 1920 h 1080 type 7
input image 1920x1080 stride 1920x1088 format=2560
resize with RGA!
rga_api version 1.10.1_[0]
once run use 22.607000 ms
post process config: box_conf_threshold = 0.25, nms_threshold = 0.45
loadLabelName ./model/coco_80_labels_list.txt
train @ (150 94 1494 1054) 0.241569
chn 0 size 243920 qp 13
time_gap=-32decoder require buffer w:h [1920:1080] stride [1920:1088] buf_size 4177920 pts=0 dts=0 get one frame 1745683729736 data_vir=0x7f97c04000 fd=27 input image 1920x1080 stride 1920x1088 format=2560
resize with RGA!
once run use 17.104000 ms
post process config: box_conf_threshold = 0.25, nms_threshold = 0.45
train @ (42 54 1449 1068) 0.300684
chn 0 size 47709 qp 18
time_gap=6decoder require buffer w:h [1920:1080] stride [1920:1088] buf_size 4177920 pts=0 dts=0 get one frame 1745683729769 data_vir=0x7f97808000 fd=22 input image 1920x1080 stride 1920x1088 format=2560
.......
会对每一帧循环打印 直到处理完成 得到 out.h264

性能
cpu 15%
npu core0: 45% core1: 0% core2: 0%
rga 6%
推理每帧耗时20ms左右

ffmpeg将h264处理成mp4
ffmpeg -i out.h264 -c:v copy -f mp4 xxx.mp4

播放视频
https://www.bilibili.com/video/BV135GSzgEs3/

可以看出 输出的视频检测到了目标画了框

处理过程
先读取h264解码出nv12
然后用rga将nv12转成rgb24 同时缩放图像宽高成模型的宽高
传给npu推理得到坐标 类别 置信度等信息
再拷贝一份原先解码出来的nv12 根据推理输出的坐标信息画框 这里画框没有用opencv和rga 而是直接根据nv12的排列方式直接修改图像数据
最后将画了框的nv12编码成h264输出