10.K3588本地大模型精度评估优化
1.精度评估工具
瑞芯微官方给出了一个精度评估工具
rknn.accuracy_analysis(inputs, output_dir, target=None, device_id=None)
生成浮点参考输出和量化输出的逐层快照，然后计算每一层的误差。
可以把它理解成三件事：
- 给模型喂一张或几张代表性输入
- 保存每一层的 FP32 参考输出和 INT8 输出
- 生成每层误差表，例如 single cos / entire cos
这里有一个参数特别重要：target=None
一般表示在 PC simulator 上做分析。它适合回答：量化模型在工具链模拟器里和浮点模型差多少？
如果设置具体芯片目标，并连接了设备，例如：
target=具体芯片平台
device_id=设备 ID
它更接近回答：真实 NPU 每一层输出和参考输出差多少？
所以逐层分析也可以分成两层：
- PC simulator 逐层分析：先定位量化和图转换问题
- 板端 NPU 逐层分析：再定位真实硬件 runtime 问题
2.连板分析报错
在连板推理时，遇到一个报错
E RKNNAPI: rknn_init, server connect fail! ret = -9(ERROR_PIPE)!
解决方法： 这通常是由于Debian固件上的adbd程序没有监听5037端口导致的，可以在板子上执行以下命令来判断:
netstat -n -t -u -a
如果输出结果中没有5037端口,则执行下列命令下载和更新adbd程序, 并重启板子;否则,跳过下列步骤。
wget -O adbd.zip https://ftzr.zbox.filez.com/v2/delivery/data/7f0ac30dfa474892841fcb2cd29ad924/adbd.zip
unzip adbd.zip
adb push adbd/linux-aarch64/adbd /usr/bin/adbd
进入设备shell命令，增加adbd的可执行权限
adb shell "chmod +x /usr/bin/adbd"
adb reboot
重启设备后，按照启动步骤启动rknn_server服务，再次尝试连板推理。
3.连板推理例子
在rknn_model_zoo-2.0.0/examples/mobilenet/python 目录进行测试
目录文件有mobilenet.py，内容如下：
import argparse
import os
import sys
import urllib
import urllib.request
import time
import traceback
import numpy as np
import cv2
from rknn.api import RKNN
from scipy.special import softmax
DATASET_PATH = '../../../datasets/imagenet/ILSVRC2012_img_val_samples/dataset_20.txt'
MODEL_DIR = '../model/'
MODEL_PATH = MODEL_DIR + 'mobilenetv2-12.onnx'
OUT_RKNN_PATH = MODEL_DIR + 'mobilenet_v2.rknn'
CLASS_LABEL_PATH = MODEL_DIR + 'synset.txt'
RKNPU1_TARGET = ['rk1808', 'rv1109', 'rv1126']
def readable_speed(speed):
speed_bytes = float(speed)
speed_kbytes = speed_bytes / 1024
if speed_kbytes > 1024:
speed_mbytes = speed_kbytes / 1024
if speed_mbytes > 1024:
speed_gbytes = speed_mbytes / 1024
return "{:.2f} GB/s".format(speed_gbytes)
else:
return "{:.2f} MB/s".format(speed_mbytes)
else:
return "{:.2f} KB/s".format(speed_kbytes)
def show_progress(blocknum, blocksize, totalsize):
speed = (blocknum * blocksize) / (time.time() - start_time)
speed_str = " Speed: {}".format(readable_speed(speed))
recv_size = blocknum * blocksize
f = sys.stdout
progress = (recv_size / totalsize)
progress_str = "{:.2f}%".format(progress * 100)
n = round(progress * 50)
s = ('#' * n).ljust(50, '-')
f.write(progress_str.ljust(8, ' ') + '[' + s + ']' + speed_str)
f.flush()
f.write('\r\n')
def check_and_download_origin_model():
global start_time
if not os.path.exists(MODEL_PATH):
print('--> Download {}'.format(MODEL_PATH))
url = 'https://ftrg.zbox.filez.com/v2/delivery/data/95f00b0fc900458ba134f8b180b3f7a1/examples/MobileNet/mobilenetv2-12.onnx'
download_file = MODEL_PATH
try:
start_time = time.time()
urllib.request.urlretrieve(url, download_file, show_progress)
except:
print('Download {} failed.'.format(download_file))
print(traceback.format_exc())
exit(-1)
print('done')
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='MobileNet Python Demo', add_help=True)
parser.add_argument('--target', type=str,
default='rk3566', help='RKNPU target platform')
parser.add_argument('--npu_device_test', action='store_true',
default=False, help='Connected npu device run')
parser.add_argument('--accuracy_analysis', action='store_true',
default=False, help='Accuracy analysis')
parser.add_argument('--eval_perf', action='store_true',
default=False, help='Time consuming evaluation')
parser.add_argument('--eval_memory', action='store_true',
default=False, help='Memory evaluation')
parser.add_argument('--model', type=str,
default=MODEL_PATH, help='onnx model path')
parser.add_argument('--output_path', type=str,
default=OUT_RKNN_PATH, help='output rknn model path')
parser.add_argument('--dtype', type=str, default='i8',
help='dtype of model, i8/fp32 for RKNPU2, u8/fp32 for RKNPU1')
args = parser.parse_args()
# Download model if not exist (from https://ftrg.zbox.filez.com/v2/delivery/data/95f00b0fc900458ba134f8b180b3f7a1/examples/MobileNet/mobilenetv2-12.onnx)
check_and_download_origin_model()
# Create RKNN object
rknn = RKNN(verbose=False)
# Pre-process config
print('--> Config model')
rknn.config(mean_values=[[255*0.485, 255*0.456, 255*0.406]], std_values=[[
255*0.229, 255*0.224, 255*0.225]], target_platform=args.target)
print('done')
# Load model
print('--> Loading model')
if args.target in RKNPU1_TARGET:
ret = rknn.load_onnx(model=args.model, inputs=['input'], input_size_list=[[3,224,224]])
else:
ret = rknn.load_onnx(model=args.model, inputs=['input'], input_size_list=[[1,3,224,224]])
if ret != 0:
print('Load model failed!')
exit(ret)
print('done')
# Build model
print('--> Building model')
do_quant = True if (args.dtype == 'i8' or args.dtype == 'u8') else False
ret = rknn.build(do_quantization=do_quant, dataset=DATASET_PATH)
if ret != 0:
print('Build model failed!')
exit(ret)
print('done')
# Export rknn model
print('--> Export rknn model')
ret = rknn.export_rknn(args.output_path)
if ret != 0:
print('Export rknn model failed!')
exit(ret)
print('done')
# Set inputs
img = cv2.imread('../model/bell.jpg')
img = cv2.resize(img, (224, 224))
img = np.expand_dims(img, 0)
# Init runtime environment
print('--> Init runtime environment')
if args.npu_device_test or args.target in RKNPU1_TARGET:
# For RKNPU1, the simulator has beed disabled since version 1.7.5
ret = rknn.init_runtime(target=args.target)
elif args.eval_perf or args.eval_memory:
ret = rknn.init_runtime(
target=args.target, perf_debug=True, eval_mem=True)
else:
if args.target in RKNPU1_TARGET:
print('The target {} does not support simulator.'.format(args.target))
print('Please set `--npu_device_test` to init runtime with real target.')
exit(-1)
ret = rknn.init_runtime()
if ret != 0:
print('Init runtime environment failed!')
exit(ret)
print('done')
# Eval Perf
if args.eval_perf:
print('--> Eval Perf')
rknn.eval_perf()
print('done')
# Eval Memory
if args.eval_memory:
print('--> Eval Memory')
rknn.eval_memory()
print('done')
# Inference
print('--> Running model')
outputs = rknn.inference(inputs=[img])
# Post Process
print('--> PostProcess')
with open(CLASS_LABEL_PATH, 'r') as f:
labels = [l.rstrip() for l in f]
scores = softmax(outputs[0])
# print the top-5 inferences class
scores = np.squeeze(scores)
a = np.argsort(scores)[::-1]
print('-----TOP 5-----')
for i in a[0:5]:
print('[%d] score=%.2f class="%s"' % (i, scores[i], labels[i]))
print('done')
# Accuracy analysis
if args.accuracy_analysis:
print('--> Accuracy analysis')
if args.npu_device_test:
ret = rknn.accuracy_analysis(
inputs=['../model/bell.jpg'], target=args.target)
else:
ret = rknn.accuracy_analysis(inputs=['../model/bell.jpg'])
if ret != 0:
print('Accuracy analysis failed!')
exit(ret)
print('done')
# Release
rknn.release()
测试指令如下
python mobilenet.py --target rk3588 --accuracy_analysis --npu_device_test
精度获取结果如下：
layer_name simulator_error runtime_error
entire single entire single_sim
cos euc cos euc cos euc cos euc
---------------------------------------------------------------------------------------------------------------
[Input] input 1.00000 | 0.0 1.00000 | 0.0
[exDataConvert] input_int8 0.99999 | 2.0815 0.99999 | 2.0815
[Conv] 474
[Clip] 317 0.99995 | 3.0455 0.99995 | 3.0455 0.99993 | 3.4390 0.99994 | 3.2604
[Conv] 477
[Clip] 320 0.99951 | 18.696 0.99954 | 18.145 0.99951 | 18.738 1.00000 | 1.5393
[Conv] 480 0.99836 | 24.715 0.99976 | 9.4394 0.99835 | 24.763 1.00000 | 0.6997
[Conv] 483
[Clip] 325 0.99860 | 23.443 0.99977 | 9.4888 0.99859 | 23.477 1.00000 | 1.0993
[Conv] 486
维度分组
- cos（余弦相似度）
-
取值范围 [-1, 1]，越接近 1 → 量化后特征和原始浮点越一致，精度越好
-
分类网络通用评判标准：
-
≥0.999：几乎无损失，量化效果优秀
-
0.99~0.999：轻微精度下降，不影响 Top5 分类
-
<0.99：明显特征失真，大概率识别掉点
- euc（欧氏距离）
- 向量空间绝对差值和，越接近 0 越好
- 数值越大，特征向量偏移越严重，量化失真越大
- MobileNet 这类分类：euc <10 基本无感知；>15 就要重点关注
4.实际分析
1.Input 转 int8 层已有基础误差 2.08，量化输入带来初始偏差；
校准集扩充到 300~500 张，包含业务场景明暗、边缘极值样本；
2.对高误差层做混合量化（FP16 保护）
表格里 Clip320、Conv480、Clip325 三层误差爆炸，单独设置为 FP16 不做 INT8 量化：
#编译时指定敏感层保持FP16
rknn.build(
do_quantization=True,
dataset="./calib.txt",
fp16_layer_list=["Conv480", "Clip320", "Clip325"]
)
中心思想是：即把误差大的网络层，从量化层转到非量化层
浙公网安备 33010602011771号