1200字范文 > 35 ubuntu20.04搭建瑞芯微的npu仿真环境和测试rv1126的Debain系统下的yolov5+npu检测

35 ubuntu20.04搭建瑞芯微的npu仿真环境和测试rv1126的Debain系统下的yolov5+npu检测

时间：2022-04-18 18:16:38

基本思想：手中有一块core-1126/1109-JD4,记录一下其刷机过程和开发人数统计，与树莓派的nanodet 每帧200ms对比一下

第一步：刷机，真的是难，各种各样的小问题，反正成功的方法只有一个，遇到问题，断电重新进行连接，刷机，基本都能解决上次的问题，这说的我都无语了,刷的debain10系统

这里注意是otg接口外侧边缘的usb接口

刷机固件网址和方法参考官方手册，附录固件升级

工具是：F:\rv1126\RKDevTool\RKDevTool_Release_v2.86

镜像是：F:\rv1126\AIO-RV1126(1109)-JD4\Buildroot\薪火车牌识别固件\AIO-RV1126_RV1109-XHLPR_IPC__0906_1125

替换配置是：F:\rv1126\Debian10\debian10__0429_1902

第二步：插上网线，进行ip地址搜索

在window的cmd命令行中，输入搜索命令

C:\Users\Administrator>for /L %i IN (1,1,254) DO ping -w 2 -n 1 192.168.10.%i

显示搜索同一网络下的ip地址

C:\Users\Administrator>arp -aInterface: 192.168.10.155 --- 0x9Internet AddressPhysical AddressType192.168.10.1ff-ff-ff-ff-ff-ffdynamic192.168.10.117 ff-ff-ff-ff-ff-ffdynamic192.168.10.134 ff-ff-ff-ff-ff-ffdynamic192.168.10.210 ff-ff-ff-ff-ff-ffdynamic192.168.10.255 ff-ff-ff-ff-ff-ffstatic224.0.0.2 01-00-5e-00-00-02static224.0.0.22 01-00-5e-00-00-16static224.0.0.251 01-00-5e-00-00-fbstatic224.0.0.252 01-00-5e-00-00-fcstatic239.255.255.250 ff-ff-ff-ff-ff-ffstatic255.255.255.255 ff-ff-ff-ff-ff-ffstatic

然后进行ssh远程连接用户名:firefly密码:firefly使用方法 — Firefly Wiki

ubuntu@sxj731533730:~$ ssh firefly@192.168.10.210firefly@192.168.10.210's password:Linux firefly 4.19.111 #1 SMP PREEMPT Thu Jul 8 17:38:26 CST armv7lThe programs included with the Debian GNU/Linux system are free software;the exact distribution terms for each program are described in theindividual files in /usr/share/doc/*/copyright.Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extentpermitted by applicable law.firefly@firefly:~$

第三步：参考手册配置,参考附录RKNN使用

补充：官方支持了python3.8，可以不用安装conda环境

RKNN-Toolkit Version 1.7.3已正式发布，请大家尽量使用最新版本！

/rockchip-linux/rknn-toolkit

/rockchip-linux/rknpu

GitHub - airockchip/RK3399Pro_npu

OP支持情况：

/rockchip-linux/rknn-toolkit/blob/master/doc/RKNN_OP_Support_V1.7.3.md

主要更新：

1. 功能优化：

精度分析接口打印各层精度信息时显示量化类型。

完善图优化规则。

2. 完善op支持，给出op限制表格。

3. 移除对Ubuntu 16.04 / Python3.5的支持，增加对Ubuntu20.04 / Python3.8的支持。

4. 升级部分依赖模块。

5. 修复已知bug

本博客写的早，所以仍然安装conda环境和配置输出

firefly@firefly:~$ sudo apt-get update && sudo apt-get install python3-pipfirefly@firefly:~$ pip3 install numpy==1.16.3 psutil==5.6.2 ruamel.yaml==0.15.81 -i https://pypi.tuna./simplefirefly@firefly:~$ sudo apt-get install wget git cmakefirefly@firefly:~$ sudo apt-get install multiarch-supportfirefly@firefly:~$ wget /debian-security/pool/updates/main/j/jasper/libjasper1_1.900.1-debian1-2.4+deb8u6_armhf.debfirefly@firefly:~$ sudo dpkg -i libjasper1_1.900.1-debian1-2.4+deb8u6_armhf.debfirefly@firefly:~$ wget /debian-security/pool/updates/main/j/jasper/libjasper-dev_1.900.1-debian1-2.4+deb8u6_armhf.debfirefly@firefly:~$ sudo dpkg -i libjasper-dev_1.900.1-debian1-2.4+deb8u6_armhf.debfirefly@firefly:~$ sudo apt-get install libhdf5-devfirefly@firefly:~$ sudo apt-get install libatlas-base-devfirefly@firefly:~$ sudo apt-get install libqtgui4firefly@firefly:~$ sudo apt-get install libqt4-testfirefly@firefly:~$ git clone /rockchip-linux/rknn-toolkit.gitfirefly@firefly:~$ wget /simple/opencv-python/opencv_python-4.0.1.24-cp37-cp37m-linux_armv7l.whlfirefly@firefly:~$ pip3 install opencv_python-4.0.1.24-cp37-cp37m-linux_armv7l.whlProcessing ./opencv_python-4.0.1.24-cp37-cp37m-linux_armv7l.whlRequirement already satisfied: numpy>=1.16.2 in ./.local/lib/python3.7/site-packages (from opencv-python==4.0.1.24) (1.16.3)Installing collected packages: opencv-pythonSuccessfully installed opencv-python-4.0.1.24firefly@firefly:~/rknn-toolkit/rknn-toolkit-lite/packages$ pip3 install rknn_toolkit_lite-1.7.1-cp37-cp37m-linux_armv7l.whlProcessing ./rknn_toolkit_lite-1.7.1-cp37-cp37m-linux_armv7l.whlRequirement already satisfied: ruamel.yaml==0.15.81 in /home/firefly/.local/lib/python3.7/site-packages (from rknn-toolkit-lite==1.7.1) (0.15.81)Requirement already satisfied: numpy==1.16.3 in /home/firefly/.local/lib/python3.7/site-packages (from rknn-toolkit-lite==1.7.1) (1.16.3)Requirement already satisfied: psutil==5.6.2 in /home/firefly/.local/lib/python3.7/site-packages (from rknn-toolkit-lite==1.7.1) (5.6.2)Installing collected packages: rknn-toolkit-liteSuccessfully installed rknn-toolkit-lite-1.7.1firefly@firefly:~$ python3Python 3.7.3 (default, Jan 22 , 20:04:44)[GCC 8.3.0] on linuxType "help", "copyright", "credits" or "license" for more information.>>> from rknnlite.api import RKNNLite>>>

第四步：配置pc端转,支持模型转rknnn和生成模型移植到板子上使用RKNNlite调用 rknn_toolkit :Index of /pypi/simple/

ubuntu@ubuntu:~$ wget https://mirrors.tuna./anaconda/miniconda/Miniconda3-latest-Linux-x86_64.shubuntu@ubuntu:~$ bash Miniconda3-latest-Linux-x86_64.sh ubuntu@ubuntu:~$ source ~/.bashrcubuntu@ubuntu:~$ conda config --set auto_activate_base falseubuntu@ubuntu:~$ conda config --add channels https://mirrors.tuna./anaconda/pkgs/free/ubuntu@ubuntu:~$ conda config --add channels https://mirrors.tuna./anaconda/pkgs/main/ubuntu@ubuntu:~$ conda config --set show_channel_urls yesubuntu@ubuntu:~$ conda create -n rknnpy36 python=3.6ubuntu@ubuntu:~$ conda activate rknnpy36(rknnpy36) ubuntu@ubuntu:~$ git clone /rockchip-linux/rknn-toolkit.git(rknnpy36) ubuntu@ubuntu:~/Downloads/rknn-toolkit-master/packages$ pip3 install -r requirements-cpu.txt -i https://pypi.tuna./simple(rknnpy36) ubuntu@ubuntu:~/Downloads/rknn-toolkit-master/packages$ pip3 install -r requirements-cpu.txt(rknnpy36) ubuntu@ubuntu:~/Downloads/rknn-toolkit-master/rknn-toolkit-lite/packages$ (rknnpy36) ubuntu@ubuntu:~/rknn-toolkit/rknn-toolkit-lite/packages$ pip3 install rknn_toolkit_lite-1.6.1-cp36-cp36m-linux_x86_64.whl -i https://pypi.tuna./simpleLooking in indexes: https://pypi.tuna./simpleProcessing ./rknn_toolkit_lite-1.7.1-cp36-cp36m-linux_x86_64.whlRequirement already satisfied: ruamel.yaml==0.15.81 in /home/ubuntu/miniconda3/envs/rknnpy36/lib/python3.6/site-packages (from rknn-toolkit-lite==1.7.1) (0.15.81)Requirement already satisfied: numpy==1.16.3 in /home/ubuntu/miniconda3/envs/rknnpy36/lib/python3.6/site-packages (from rknn-toolkit-lite==1.7.1) (1.16.3)Requirement already satisfied: psutil==5.6.2 in /home/ubuntu/miniconda3/envs/rknnpy36/lib/python3.6/site-packages (from rknn-toolkit-lite==1.7.1) (5.6.2)Installing collected packages: rknn-toolkit-liteSuccessfully installed rknn-toolkit-lite-1.7.1(rknnpy36) ubuntu@ubuntu:~/Downloads$ wget http://repo.rock-/pypi/simple/rknn-toolkit/rknn_toolkit-1.7.1-cp36-cp36m-linux_x86_64.whl(rknnpy36) ubuntu@ubuntu:~/rknn-toolkit/rknn-toolkit-lite/packages$ pip3 install rknn_toolkit-1.7.1-cp36-cp36m-linux_x86_64.whl -i https://pypi.tuna./simple(rknnpy36) ubuntu@ubuntu:~/Downloads$ python3Python 3.6.13 |Anaconda, Inc.| (default, Jun 4 , 14:25:59) [GCC 7.5.0] on linuxType "help", "copyright", "credits" or "license" for more information.>>> from rknn.api import RKNN>>> from rknnlite.api import RKNNLite>>>

第五步先测试一下仿真环境，以yolov5测试

ubuntu@ubuntu:~$ source ~/.bashrc ubuntu@ubuntu:~$ conda activate rknnpy36(rknnpy36) ubuntu@ubuntu:~$ cd rknn-toolkit/examples/onnx/yolov5(rknnpy36) ubuntu@ubuntu:~/rknn-toolkit/examples/onnx/yolov5$ python3 test.py --> Config modeldone--> Loading modelW Call onnx.optimizer.optimize fail, skip optimizeW Please confirm that your onnx opset_version <= 11 (current opset_verison = 12)!!!

生成模型

模型生成和测试成功

第六步：使用转的模型测试rv1126在开发板上执行

firefly@firefly:~/rknn-toolkit/rknn-toolkit-lite/examples/inference_with_lite$ python3 test.py--> Load RKNN modeldone--> Init runtime environmentE [vnn_CreateRKNN:5512]rknn model target platform[0] is RK1808, while current platform is RK1109Create RKNN model fail, error=-13E Catch exception when init runtime!E Traceback (most recent call last):File "/home/firefly/.local/lib/python3.7/site-packages/rknnlite/api/rknn_lite.py", line 157, in init_runtimeself.rknn_runtime.build_graph(self.rknn_data, self.load_model_in_npu)File "rknnlite/api/rknn_runtime.py", line 733, in rknnlite.api.rknn_runtime.RKNNRuntime.build_graphException: RKNN init failed. error code: RKNN_ERR_TARGET_PLATFORM_UNMATCHInit runtime environment failed

我测试过程中，发现官方模型不适合我的开发板，所以修改yolov5代码中的

rknn.config(reorder_channel='0 1 2',mean_values=[[0, 0, 0]],std_values=[[255, 255, 255]],optimization_level=3,target_platform = 'rk1808',output_optimize=1,quantize_input_node=QUANTIZE_ON)

修改之后，重新生成模型

rknn.config(reorder_channel='0 1 2',mean_values=[[0, 0, 0]],std_values=[[255, 255, 255]],optimization_level=3,target_platform = 'rv1126',output_optimize=1,quantize_input_node=QUANTIZE_ON)

测试环境

firefly@firefly:~/yolov5$ dpkg -l | grep npuii libxi6:armhf2:1.7.9-1 armhf X11 Input extension libraryii rknpu 1.6.0armhf <rockchip rknpu package>

测试代码官方代码改的

import osimport urllibimport tracebackimport timeimport sysimport numpy as npimport cv2from rknnlite.api import RKNNLiteimport timeONNX_MODEL = 'yolov5s.onnx'RKNN_MODEL = 'yolov5s.rknn'IMG_PATH = 'bus.jpg'DATASET = 'dataset.txt'QUANTIZE_ON = TrueBOX_THRESH = 0.5NMS_THRESH = 0.6IMG_SIZE = 640CLASSES = ("person", "bicycle", "car","motorbike ","aeroplane ","bus ","train","truck ","boat","traffic light","fire hydrant","stop sign ","parking meter","bench","bird","cat","dog ","horse ","sheep","cow","elephant","bear","zebra ","giraffe","backpack","umbrella","handbag","tie","suitcase","frisbee","skis","snowboard","sports ball","kite","baseball bat","baseball glove","skateboard","surfboard","tennis racket","bottle","wine glass","cup","fork","knife ","spoon","bowl","banana","apple","sandwich","orange","broccoli","carrot","hot dog","pizza ","donut","cake","chair","sofa","pottedplant","bed","diningtable","toilet ","tvmonitor","laptop","mouse","remote ","keyboard ","cell phone","microwave ","oven ","toaster","sink","refrigerator ","book","clock","vase","scissors ","teddy bear ","hair drier", "toothbrush ")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2 # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2 # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2 # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2])*2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE/grid_h)box_wh = pow(sigmoid(input[..., 2:4])*2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""box_classes = np.argmax(box_class_probs, axis=-1)box_class_scores = np.max(box_class_probs, axis=-1)pos = np.where(box_confidences[...,0] >= BOX_THRESH)boxes = boxes[pos]classes = box_classes[pos]scores = box_class_scores[pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input,mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2] # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh paddingdw /= 2 # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad: # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add borderreturn im, ratio, (dw, dh)if __name__ == '__main__':# Create RKNN objectrknn = RKNNLite()# load RKNN modelprint('--> Load RKNN model')ret = rknn.load_rknn('./yolov5s.rknn')if ret != 0:print('Load RKNN model failed')exit(ret)print('done')# init runtime environmentprint('--> Init runtime environment')#ret = rknn.init_runtime()ret = rknn.init_runtime('rv1126', device_id='rv1126')if ret != 0:print('Init runtime environment failed')exit(ret)print('done')start = time.time()# Set inputsimg = cv2.imread(IMG_PATH)# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = cv2.resize(img,(IMG_SIZE, IMG_SIZE))# Inferenceprint('--> Running model')outputs = rknn.inference(inputs=[img])# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3,-1]+list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3,-1]+list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3,-1]+list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)if boxes is not None:draw(img_1, boxes, scores, classes)end = time.time()print (end-start,"s")cv2.imwrite("result.jpg", img_1)rknn.release()

测试结果

firefly@firefly:~/yolov5$ sudo python3 test.py--> Load RKNN modeldone--> Init runtime environmentdone--> Running modelclass: person, score: 0.9981504082679749box coordinate left,top,right,down: [479.5712496638298, 257.1674613952637, 560.4287503361702, 516.2188911437988]class: person, score: 0.9968554973602295box coordinate left,top,right,down: [110.34747083485126, 229.37330585718155, 220.6373619288206, 531.1496433615685]class: person, score: 0.9822962880134583box coordinate left,top,right,down: [209.4318208694458, 244.13265788555145, 287.1548795700073, 506.1707478761673]class: bus , score: 0.9902657866477966box coordinate left,top,right,down: [90.29679995775223, 138.5124049782753, 552.5043475031853, 440.2887424826622]0.619182825088501 s

方法一：

第七步：我对比了官方模型和yolov5的版本，感觉最像yolov5 5.0版本，所以训练一个看看32 、 YOLO5训练自己的模型以及转ncnn模型_sxj731533730的博客-CSDN博客

ubuntu@ubuntu:~/yolov5-5.0$ python3 train.py --img 640 --batch 16 --epoch 10 --data data/trainData.yaml --cfg models/yolov5s.yaml --weights weights/yolov5s.pt --device "0"ubuntu@ubuntu:~/yolov5-5.0$ python3 export.py --weights runs/train/exp3/weights/best.pt --img 640 --batch 1

转模型成功

from rknn.api import RKNNONNX_MODEL = '/home/ubuntu/yolov5-5.0/runs/train/exp3/weights/best.onnx'RKNN_MODEL = 'yolov5s.rknn'if __name__ == '__main__':# Create RKNN objectrknn = RKNN(verbose=True)# pre-process configprint('--> config model')rknn.config(mean_values=[[0, 0, 0]], std_values=[[255, 255, 255]], reorder_channel='0 1 2',target_platform='rv1126',quantized_dtype='asymmetric_affine-u8', optimization_level=3, output_optimize=1)print('done')print('--> Loading model')ret = rknn.load_onnx(model=ONNX_MODEL)if ret != 0:print('Load model failed!')exit(ret)print('done')# Build modelprint('--> Building model')ret = rknn.build(do_quantization=True, dataset='/home/ubuntu/yolov5-5.0/dataset.txt') # ,pre_compile=Trueif ret != 0:print('Build yolov5s failed!')exit(ret)print('done')# Export rknn modelprint('--> Export RKNN model')ret = rknn.export_rknn(RKNN_MODEL)if ret != 0:print('Export yolov5s.rknn failed!')exit(ret)print('done')rknn.release()

但是测试失败....,然后对比了两个onnx文件，感觉应该修改onnx的输出，然后果然发现了志同道合的大佬,参考附录rv1126

修改一下/home/ubuntu/yolov5-5.0/models/yolo.py，重新生成onnx，我差点去修改onnx，在重新生成一下onnx

x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()修改为x[i] = x[i].view(bs, self.na, self.no, ny, nx)

然后在转rknn,量化图片自己搞,最后大佬贴心的还提供了检测代码，稍微修改

import osimport urllibimport tracebackimport timeimport sysimport numpy as npimport cv2from rknn.api import RKNNRKNN_MODEL = '/home/ubuntu/yolov5-5.0/yolov5s.rknn'IMG_PATH = 'dog.jpg'QUANTIZE_ON = TrueBOX_THRESH = 0.5NMS_THRESH = 0.6IMG_SIZE = 640CLASSES = ("drug","battery","apple","bottle","banana","smoke","ylg")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2 # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2 # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2 # bottom right yreturn ydef resize_postprocess(x, offset_x, offset_y):# Convert [x1, y1, x2, y2] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] / offset_x # top left xy[:, 1] = x[:, 1] / offset_y # top left yy[:, 2] = x[:, 2] / offset_x # bottom right xy[:, 3] = x[:, 3] / offset_y # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2]) * 2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE / grid_h)box_wh = pow(sigmoid(input[..., 2:4]) * 2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""box_classes = np.argmax(box_class_probs, axis=-1)box_class_scores = np.max(box_class_probs, axis=-1)pos = np.where(box_confidences[..., 0] >= BOX_THRESH)boxes = boxes[pos]classes = box_classes[pos]scores = box_class_scores[pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input, mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2] # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh paddingdw /= 2 # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad: # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add borderreturn im, ratio, (dw, dh)def letter_box_postprocess(x, scalingfactor, xy_correction):y = np.copy(x)y[:, 0] = (x[:, 0] - xy_correction[0]) / scalingfactor # top left xy[:, 1] = (x[:, 1] - xy_correction[1]) / scalingfactor # top left yy[:, 2] = (x[:, 2] - xy_correction[0]) / scalingfactor # bottom right xy[:, 3] = (x[:, 3] - xy_correction[1]) / scalingfactor # bottom right yreturn ydef get_file(filepath):templist = []with open(filepath, "r") as f:for item in f.readlines():templist.append(item.strip())return templistif __name__ == '__main__':# Create RKNN objectrknn = RKNN()image_process_mode = "letter_box"print("image_process_mode = ", image_process_mode)if not os.path.exists(RKNN_MODEL):print('model not exist')exit(-1)# Load ONNX modelprint('--> Loading model')ret = rknn.load_rknn(RKNN_MODEL)if ret != 0:print('Load rknn model failed!')exit(ret)print('done')# init runtime environmentprint('--> Init runtime environment')ret = rknn.init_runtime()# ret = rknn.init_runtime('rk180_8', device_id='1808')if ret != 0:print('Init runtime environment failed')exit(ret)print('done')image = cv2.imread("/home/ubuntu/yolov5-5.0/trainData/images/dc1_02.jpg")img_height = image.shape[0]img_width = image.shape[1]# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)if image_process_mode == "resize":img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))elif image_process_mode == "letter_box":img, scale_factor, correction = letterbox(img)# Inferenceprint('--> Running model')outputs = rknn.inference(inputs=[img])# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3, -1] + list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3, -1] + list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3, -1] + list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)if image_process_mode == "resize":scale_h = IMG_SIZE / img_heightscale_w = IMG_SIZE / img_widthboxes = resize_postprocess(boxes, scale_w, scale_h)elif image_process_mode == "letter_box":boxes = letter_box_postprocess(boxes, scale_factor[0], correction)# img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)if boxes is not None:draw(image, boxes, scores, classes)cv2.imwrite("image.jpg", image)rknn.release()

测试结果 yyds

/home/ubuntu/miniconda3/envs/rknnpy36/bin/python /home/ubuntu/yolov5-5.0/rknn_detect.pyimage_process_mode = letter_box--> Loading modeldone--> Init runtime environmentlibrknn_runtime version 1.7.1 (bd41dbc build: -10-28 16:15:23 base: 1131)done--> Running modelclass: battery, score: 0.9596196413040161box coordinate left,top,right,down: [100.91769161224364, 57.873418962955476, 129.09340457916258, 105.65749076604843]Process finished with exit code 0

开发板上实测代码，需要修改，代码中有bug,需要修改

import osimport urllibimport tracebackimport timeimport sysimport numpy as npimport cv2from rknnlite.api import RKNNLiteRKNN_MODEL = 'yolov5s.rknn'QUANTIZE_ON = TrueBOX_THRESH = 0.5NMS_THRESH = 0.6IMG_SIZE = 640CLASSES = ("drug","battery","apple","bottle","banana","smoke","ylg")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2 # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2 # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2 # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2 # bottom right yreturn ydef resize_postprocess(x, offset_x, offset_y):# Convert [x1, y1, x2, y2] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] / offset_x # top left xy[:, 1] = x[:, 1] / offset_y # top left yy[:, 2] = x[:, 2] / offset_x # bottom right xy[:, 3] = x[:, 3] / offset_y # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2]) * 2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE / grid_h)box_wh = pow(sigmoid(input[..., 2:4]) * 2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""box_classes = np.argmax(box_class_probs, axis=-1)box_class_scores = np.max(box_class_probs, axis=-1)pos = np.where(box_confidences[..., 0] >= BOX_THRESH)boxes = boxes[pos]classes = box_classes[pos]scores = box_class_scores[pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input, mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2] # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1] # wh paddingdw /= 2 # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad: # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color) # add borderreturn im, ratio, (dw, dh)def letter_box_postprocess(x, scalingfactor, xy_correction):y = np.copy(x)y[:, 0] = (x[:, 0] - xy_correction[0]) / scalingfactor # top left xy[:, 1] = (x[:, 1] - xy_correction[1]) / scalingfactor # top left yy[:, 2] = (x[:, 2] - xy_correction[0]) / scalingfactor # bottom right xy[:, 3] = (x[:, 3] - xy_correction[1]) / scalingfactor # bottom right yreturn ydef get_file(filepath):templist = []with open(filepath, "r") as f:for item in f.readlines():templist.append(item.strip())return templistif __name__ == '__main__':# Create RKNN objectrknn = RKNNLite()image_process_mode = "letter_box"print("image_process_mode = ", image_process_mode)if not os.path.exists(RKNN_MODEL):print('model not exist')exit(-1)# Load ONNX modelprint('--> Loading model')ret = rknn.load_rknn(RKNN_MODEL)if ret != 0:print('Load rknn model failed!')exit(ret)print('done')# init runtime environmentprint('--> Init runtime environment')ret = rknn.init_runtime()# ret = rknn.init_runtime('rk180_8', device_id='1808')if ret != 0:print('Init runtime environment failed')exit(ret)print('done')image = cv2.imread("0.jpg")img_height = image.shape[0]img_width = image.shape[1]# img, ratio, (dw, dh) = letterbox(img, new_shape=(IMG_SIZE, IMG_SIZE))img = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)if image_process_mode == "resize":img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))elif image_process_mode == "letter_box":img, scale_factor, correction = letterbox(img)# Inferenceprint('--> Running model')start = time.time()outputs = rknn.inference(inputs=[img])# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3, -1] + list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3, -1] + list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3, -1] + list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)if boxes is not None:if image_process_mode == "resize":scale_h = IMG_SIZE / img_heightscale_w = IMG_SIZE / img_widthboxes = resize_postprocess(boxes, scale_w, scale_h)elif image_process_mode == "letter_box":boxes = letter_box_postprocess(boxes, scale_factor[0], correction)# img_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)if boxes is not None:draw(image, boxes, scores, classes)end = time.time()print (end-start,"s")cv2.imwrite("image.jpg", image)rknn.release()

开发板实测结果

firefly@firefly:~/yolov5/yolov5_0$ sudo python3 test_rknn.pyimage_process_mode = letter_box--> Loading modeldone--> Init runtime environmentdone--> Running modelclass: battery, score: 0.9596196413040161box coordinate left,top,right,down: [100.91770471334458, 57.873415207862855, 129.09341284036637, 105.65749452114105]0.21574974060058594 s

方法二 :

第八步：发现有人写过这方面的文章，下载官方手册指导的源码的github指导和结合yolov5训练即可，参考附录如何训练yolov5

ubuntu@ubuntu:~$ git clone /littledeep/YOLOv5-RK3399Proubuntu@ubuntu:~/YOLOv5-RK3399Pro$ python3 train.py --img 640 --batch 16 --epoch 10 --data data/trainData.yaml --hyp data/hyp.scratch.yaml --cfg models/yolov5s.yaml --weights weights/yolov5s.pt --device "0"

其中的超参数配置文件和与预训练的权重都来自yolov5 v4.0

/ultralytics/yolov5/blob/v4.0/data/hyp.scratch.yaml

训练结果

Epoch gpu_mem box obj clstotal labels img_size199/1992.33G 0.01391 0.006809 0.0007568 0.02147 23 640: 100%|█| 106/106 [00:17<00:00, ClassImagesLabels P RmAP@.5 mAP@.5:.95: 100%|█| 53/53 [00:all 1692 1692 0.999 0.999 0.996 0.92drug 1692 359 1 1 0.997 0.937battery 1692 157 0.999 1 0.995 0.889apple 1692 116 0.998 1 0.996 0.959bottle 1692 358 1 1 0.996 0.898banana 1692 225 1 0.996 0.996 0.918smoke 1692 107 1 1 0.996 0.894ylg 1692 370 1 1 0.996 0.942200 epochs completed in 1.518 hours.Optimizer stripped from runs/train/exp6/weights/last.pt, 14.4MBOptimizer stripped from runs/train/exp6/weights/best.pt, 14.4MB

第九步：修改optset=11,且添加--grid 设置false

parser.add_argument('--grid', default=False, action='store_true', help='export Detect() layer grid')

然后转模型

ubuntu@ubuntu:~/YOLOv5-RK3399Pro$ cp models/export.py .ubuntu@ubuntu:~/YOLOv5-RK3399Pro$ python3 export.py --weights /home/ubuntu/YOLOv5-RK3399Pro/runs/train/exp6/weights/best.pt --rknn_mode --img 640 --batch 1

然后将onnx模型转成rknn模型，/home/ubuntu/YOLOv5-RK3399Pro/convert/rknn_convert.py代码做了细微修改，去掉远程连接开发板测试，改成本地测试（注意此时环境要切换rknnpy36）

import yamlfrom rknn.api import RKNNimport cv2_model_load_dict = {'caffe': 'load_caffe','tensorflow': 'load_tensorflow','tflite': 'load_tflite','onnx': 'load_onnx','darknet': 'load_darknet','pytorch': 'load_pytorch','mxnet': 'load_mxnet','rknn': 'load_rknn',}yaml_file = './config.yaml'def main():with open(yaml_file, 'r') as F:config = yaml.safe_load(F)# print('config is:')# print(config)model_type = config['running']['model_type']print('model_type is {}'.format(model_type))#检查模型的类型rknn = RKNN(verbose=True)#配置文件print('--> config model')rknn.config(**config['config'])print('done')print('--> Loading model')load_function = getattr(rknn, _model_load_dict[model_type])ret = load_function(**config['parameters'][model_type])if ret != 0:print('Load yolo failed! Ret = {}'.format(ret))exit(ret)print('done')#####print('hybrid_quantization')#ret = rknn.hybrid_quantization_step1(dataset=config['build']['dataset'])if model_type != 'rknn':print('--> Building model')ret = rknn.build(**config['build'])print('acc_eval')rknn.accuracy_analysis(inputs='/home/ubuntu/YOLOv5-RK3399Pro/convert/dataset 1.txt', target='rk1126')#根据个人开发板设置print('acc_eval done!')if ret != 0:print('Build yolo failed!')exit(ret)else:print('--> skip Building model step, cause the model is already rknn')#导出RKNN模型if config['running']['export'] is True:print('--> Export RKNN model')ret = rknn.export_rknn(**config['export_rknn'])if ret != 0:print('Init runtime environment failed1')exit(ret)else:print('--> skip Export model')#初始化print('--> Init runtime environment2')ret = rknn.init_runtime()if ret != 0:print('Init runtime environment failed2')exit(ret)print('done')print('--> load img')img = cv2.imread(config['img']['path'])print('img shape is {}'.format(img.shape))img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)inputs = [img]print(inputs[0][0:10,0,0])#推理if config['running']['inference'] is True:print('--> Running model')config['inference']['inputs'] = inputs#print(config['inference'])outputs = rknn.inference(inputs)#outputs = rknn.inference(config['inference'])print('len of output {}'.format(len(outputs)))print('outputs[0] shape is {}'.format(outputs[0].shape))print(outputs[0][0][0:2])else:print('--> skip inference')#评价if config['running']['eval_perf'] is True:print('--> Begin evaluate model performance')config['inference']['inputs'] = inputsperf_results = rknn.eval_perf(inputs=[img])else:print('--> skip eval_perf')if __name__ == '__main__':main()

同时需要修改同目录下的配置文件config.yaml

1)设置要转模型的onnx路径内部rk3399 pro 我一律改成rv1126

onnx:model: '/home/ubuntu/YOLOv5-RK3399Pro/runs/train/exp6/weights/best.onnx'

2)设置要进行量化的图片列表

build:do_quantization: Truedataset: '/home/ubuntu/YOLOv5-RK3399Pro/convert/dataset 1.txt' # '/home/zen/rknn_convert/quant_data/hand_dataset/pic_path_less.txt'pre_compile: False

dataset 1.txt填入要量化的图片全路径即可，多行

3)设置要输出rknn模型的路径

export_rknn:export_path: './best_noop1.rknn'

4)设置测试rknn模型的图片路径

img: &imgpath: '/home/ubuntu/YOLOv5-RK3399Pro/trainData/images/yp4_09.jpg'

5)batch_size设置大一些

config:#mean_value: [[0,0,0]]#std_value: [[58.82,58.82,58.82]]channel_mean_value: '0 0 0 255' # 123.675 116.28 103.53 58.395 # 0 0 0 255reorder_channel: '0 1 2' # '2 1 0'need_horizontal_merge: Falsebatch_size: 20epochs: -1target_platform: ['rk1126']quantized_dtype: 'asymmetric_quantized-u8'#asymmetric_quantized-u8,dynamic_fixed_point-8,dynamic_fixed_point-16optimization_level: 1

进行模型转换本菜机电脑 i5 12代 RTX3060 12G 内存16G

ubuntu@ubuntu:~/YOLOv5-RK3399Pro$ conda activate rknnpy36(rknnpy36) ubuntu@ubuntu:~/YOLOv5-RK3399Pro$ cd convert/(rknnpy36) ubuntu@ubuntu:~/YOLOv5-RK3399Pro/convert$ python3 rknn_convert.pyubuntu@ubuntu:~/YOLOv5-RK3399Pro$ conda activate rknnpy36(rknnpy36) ubuntu@ubuntu:~/YOLOv5-RK3399Pro$ cd convert/(rknnpy36) ubuntu@ubuntu:~/YOLOv5-RK3399Pro/convert$ python3 rknn_convert.py.....D 546(36.76%), Queue size 1D 547(36.82%), Queue size 1D 548(36.89%), Queue size 1D 549(36.96%), Queue size 1D 550(37.02%), Queue size 1D 551(37.09%), Queue size 1D 552(37.16%), Queue size 1D 553(37.23%), Queue size 1D 554(37.29%), Queue size 1D 555(37.36%), Queue size 1D 556(37.43%), Queue size 1D 557(37.50%), Queue size 1D 558(37.56%), Queue size 1D 559(37.63%), Queue size 1D 560(37.70%), Queue size 1....10convolution.relu.pooling.layer2_2 1434 openvx.tensor_transpose_31640convolution.relu.pooling.layer2_2 101622convolution.relu.pooling.layer2_2 32548convolution.relu.pooling.layer2_2 32533convolution.relu.pooling.layer2_2 8221convolution.relu.pooling.layer2_2 36012convolution.relu.pooling.layer2_2 2089 convolution.relu.pooling.layer2_2 983 openvx.tensor_transpose_37Total Time(us): 31431FPS(600MHz): 23.86FPS(800MHz): 31.82Note: Time of each layer is converted according to 800MHz!========================================================================

第十步：测试模型，修改i配置/home/ubuntu/YOLOv5-RK3399Pro/rknn_detect/models

class: rknn_detect_yolov5.Detectoropt:model: "/home/ubuntu/YOLOv5-RK3399Pro/convert/best_noop1.rknn"size: [ 640, 640 ]masks: [ [ 0, 1, 2 ], [ 3, 4, 5 ], [ 6, 7, 8 ] ]anchors: [ [ 10,13 ], [ 16,30 ], [ 33,23 ], [ 30,61 ], [ 62,45 ], [ 59,119 ], [ 116,90 ], [ 156,198 ], [ 373,326 ] ]names: [ "drug","battery","apple","bottle","banana","smoke","ylg" ]conf_thres: 0.3iou_thres: 0.5platform: 0

代码也是该作者的/home/ubuntu/YOLOv5-RK3399Pro/rknn_detect/rknn_detect_for_yolov5_original.py，问题不大，贴一下输入改成640

import cv2import timeimport randomimport numpy as npfrom rknn.api import RKNN"""yolov5 官方原版预测脚本 for rknn"""def get_max_scale(img, max_w, max_h):h, w = img.shape[:2]scale = min(max_w / w, max_h / h, 1)return scaledef get_new_size(img, scale):return tuple(map(int, np.array(img.shape[:2][::-1]) * scale))def sigmoid(x):return 1 / (1 + np.exp(-x))def filter_boxes(boxes, box_confidences, box_class_probs, conf_thres):box_scores = box_confidences * box_class_probs # 条件概率，在该cell存在物体的概率的基础上是某个类别的概率box_classes = np.argmax(box_scores, axis=-1) # 找出概率最大的类别索引box_class_scores = np.max(box_scores, axis=-1) # 最大类别对应的概率值pos = np.where(box_class_scores >= conf_thres) # 找出概率大于阈值的item# pos = box_class_scores >= OBJ_THRESH # 找出概率大于阈值的itemboxes = boxes[pos]classes = box_classes[pos]scores = box_class_scores[pos]return boxes, classes, scoresdef nms_boxes(boxes, scores, iou_thres):x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2]h = boxes[:, 3]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= iou_thres)[0]order = order[inds + 1]keep = np.array(keep)return keepdef plot_one_box(x, img, color=None, label=None, line_thickness=None):tl = line_thickness or round(0.002 * (img.shape[0] + img.shape[1]) / 2) + 1 # line/font thicknesscolor = color or [random.randint(0, 255) for _ in range(3)]c1, c2 = (int(x[0]), int(x[1])), (int(x[2]), int(x[3]))cv2.rectangle(img, c1, c2, color, thickness=tl, lineType=cv2.LINE_AA)if label:tf = max(tl - 1, 1) # font thicknesst_size = cv2.getTextSize(label, 0, fontScale=tl / 3, thickness=tf)[0]c2 = c1[0] + t_size[0], c1[1] - t_size[1] - 3cv2.rectangle(img, c1, c2, color, -1, cv2.LINE_AA) # filledcv2.putText(img, label, (c1[0], c1[1] - 2), 0, tl / 3, [225, 255, 255], thickness=tf, lineType=cv2.LINE_AA)print(label)def auto_resize(img, max_w, max_h):h, w = img.shape[:2]scale = min(max_w / w, max_h / h, 1)new_size = tuple(map(int, np.array(img.shape[:2][::-1]) * scale))return cv2.resize(img, new_size), scaledef letterbox(img, new_wh=(640, 640), color=(114, 114, 114)):new_img, scale = auto_resize(img, *new_wh)shape = new_img.shapenew_img = cv2.copyMakeBorder(new_img, 0, new_wh[1] - shape[0], 0, new_wh[0] - shape[1], cv2.BORDER_CONSTANT,value=color)return new_img, (new_wh[0] / scale, new_wh[1] / scale)def load_model(model_path, npu_id):rknn = RKNN()print('-->loading model : ' + model_path)rknn.load_rknn(model_path)ret = rknn.init_runtime()if ret != 0:print('Init runtime environment failed')exit(ret)print('done')return rknnclass Detector:def __init__(self, opt):self.opt = optmodel = opt['model']wh = opt['size']masks = opt['masks']anchors = opt['anchors']names = opt['names']conf_thres = opt['conf_thres']iou_thres = opt['iou_thres']platform = opt['platform']self.wh = whself.size = whself._masks = masksself._anchors = anchorsself.names = list(filter(lambda a: len(a) > 0, map(lambda x: x.strip(), open(names, "r").read().split()))) if isinstance(names, str) else namesself.conf_thres = conf_thresself.iou_thres = iou_thresif isinstance(model, str):model = load_model(model, platform)self._rknn = modelself.draw_box = Falsedef _predict(self, img_src, img, gain):src_h, src_w = img_src.shape[:2]# _img = cv2.cvtColor(_img, cv2.COLOR_BGR2RGB)img = img[..., ::-1] #img = np.concatenate([img[::2, ::2], img[1::2, ::2], img[::2, 1::2], img[1::2, 1::2]], 2)t0 = time.time()pred_onx = self._rknn.inference(inputs=[img])print("inference time:\t", time.time() - t0)boxes, classes, scores = [], [], []for t in range(3):input0_data = sigmoid(pred_onx[t][0])input0_data = np.transpose(input0_data, (1, 2, 0, 3))grid_h, grid_w, channel_n, predict_n = input0_data.shapeanchors = [self._anchors[i] for i in self._masks[t]]box_confidence = input0_data[..., 4]box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = input0_data[..., 5:]box_xy = input0_data[..., :2]box_wh = input0_data[..., 2:4]col = np.tile(np.arange(0, grid_w), grid_h).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_w)col = col.reshape((grid_h, grid_w, 1, 1)).repeat(3, axis=-2)row = row.reshape((grid_h, grid_w, 1, 1)).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy = box_xy * 2 - 0.5 + gridbox_wh = (box_wh * 2) ** 2 * anchorsbox_xy /= (grid_w, grid_h) # 计算原尺寸的中心box_wh /= self.wh # 计算原尺寸的宽高box_xy -= (box_wh / 2.) # 计算原尺寸的中心box = np.concatenate((box_xy, box_wh), axis=-1)res = filter_boxes(box, box_confidence, box_class_probs, self.conf_thres)boxes.append(res[0])classes.append(res[1])scores.append(res[2])boxes, classes, scores = np.concatenate(boxes), np.concatenate(classes), np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s, self.iou_thres)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if len(nboxes) < 1:return [], []boxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)label_list = []box_list = []for (x, y, w, h), score, cl in zip(boxes, scores, classes):print(score)x *= gain[0]y *= gain[1]w *= gain[0]h *= gain[1]x1 = max(0, np.floor(x).astype(int))y1 = max(0, np.floor(y).astype(int))x2 = min(src_w, np.floor(x + w + 0.5).astype(int))y2 = min(src_h, np.floor(y + h + 0.5).astype(int))# label_list.append(self.names[cl])label_list.append(cl)box_list.append((x1, y1, x2, y2))if self.draw_box:plot_one_box((x1, y1, x2, y2), img_src, label=self.names[cl])return label_list, np.array(box_list)def detect_resize(self, img_src):"""预测一张图片，预处理使用resizereturn: labels,boxes"""_img = cv2.resize(img_src, self.wh)gain = img_src.shape[:2][::-1]return self._predict(img_src, _img, gain)def detect(self, img_src):"""预测一张图片，预处理保持宽高比return: labels,boxes"""_img, gain = letterbox(img_src, self.wh)return self._predict(img_src, _img, gain)def close(self):self._rknn.release()def __enter__(self):return selfdef __exit__(self, exc_type, exc_val, exc_tb):self.close()def __del__(self):self.close()def test_video(det, video_path):reader = cv2.VideoCapture()reader.open(video_path)while True:ret, frame = reader.read()if not ret:breakt0 = time.time()det.detect(frame)print("total time", time.time() - t0)cv2.imshow("res", auto_resize(frame, 1200, 600)[0])cv2.waitKey(1)if __name__ == '__main__':import yamlimport cv2image = cv2.imread("/home/ubuntu/YOLOv5-RK3399Pro/trainData/images/ylg2_098.jpg")with open("/home/ubuntu/YOLOv5-RK3399Pro/rknn_detect/models/yolov5_rknn_640x640.yaml", "rb") as f:cfg = yaml.load(f, yaml.FullLoader)d = Detector(cfg["opt"])d.draw_box = Trued.detect(image)cv2.imshow("res", image)cv2.imwrite("image.jpg",image)cv2.waitKey()cv2.destroyAllWindows()

测试结果

(rknnpy36) ubuntu@ubuntu:~/YOLOv5-RK3399Pro/rknn_detect$ python3 rknn_detect_for_yolov5_original.py -->loading model : /home/ubuntu/YOLOv5-RK3399Pro/convert/best_noop1.rknnlibrknn_runtime version 1.7.1 (bd41dbc build: -10-28 16:15:23 base: 1131)doneinference time: 62.444551706314090.85787344ylg #易拉罐-->loading model : /home/ubuntu/YOLOv5-RK3399Pro/convert/best_noop1.rknnlibrknn_runtime version 1.7.1 (bd41dbc build: -10-28 16:15:23 base: 1131)doneinference time: 64.14252042770386bananaylg

量化影响精度，可以换其他量化手段

asymmetric_quantized-u8,dynamic_fixed_point-8,dynamic_fixed_point-16

修改配置文件即可，也或许是我的图片太小了，或者其它问题，下一篇使用c++修改官方demo

贴个结果2g+16g

firefly@firefly:~/sxj_client/build$ sudo ./sxj_clientsdk version: librknn_runtime version 1.6.0 (6523e57 build: -01-15 15:56:31 base: 1126) driver version: 6.4.3.5.293908model input num: 1, output num: 3index=0 name=images_220 n_dims=4 dims=[1 3 640 640] n_elems=1228800 size=2457600 fmt=0 type=1 qnt_type=0 fl=-65 zp=191 scale=0.073182index=0 name=Reshape_Reshape_259/out0_0 n_dims=5 dims=[3 85 80 80] n_elems=1632000 size=1632000 fmt=0 type=3 qnt_type=2 fl=-56 zp=200 scale=0.088509index=1 name=Reshape_Reshape_274/out0_1 n_dims=5 dims=[3 85 40 40] n_elems=408000 size=408000 fmt=0 type=3 qnt_type=2 fl=-69 zp=187 scale=0.081693index=2 name=Reshape_Reshape_289/out0_2 n_dims=5 dims=[3 85 20 20] n_elems=102000 size=102000 fmt=0 type=3 qnt_type=2 fl=-65 zp=191 scale=0.073182model is NCHW input fmtinput_width=640 input_height=640model input height=640, width=640, channel=3img.cols: 640, img.rows: 640171.16 msdog @ (279 18 458 216) 0.492511cat @ (176 103 605 406) 0.413417

36、rk3399 pro 环境搭建和yolov5 c++修改使用opencv开发使用_sxj731533730的博客-CSDN博客

十、视频流拉海康摄像头，进行mpp的c++硬解码h264编码，

firefly@firefly:~/mpp_rtsp/build$ ./rk_mpp1ms14ms1ms1ms1ms1ms310ms1ms1ms1ms1ms1ms1ms1ms1ms1ms1ms

视频流显示画面

参考：

Index of /pypi/simple/

RKNN 使用 — Firefly Wiki

使用方法 — Firefly Wiki

固件升级 — Firefly Wiki

32 、 YOLO5训练自己的模型以及转ncnn模型_sxj731533730的博客-CSDN博客

瑞芯微RV1126/1109开发流程之模型转换_xidaoliang123的博客-CSDN博客_rv1126开发

YOLOv5s部署在瑞芯微电子RK3399Pro中使用NPU进行加速推理_Deepsdu的博客-CSDN博客