『计算机视觉』Mask-RCNN_推断网络其五：目标检测结果精炼
一、Detections网络
经过了ROI网络，我们已经获取了全部推荐区域的信息，包含：
推荐区域特征（ROIAlign得到）
推荐区域类别
推荐区域坐标修正项（deltas）
再加上推荐区域原始坐标[IMAGES_PER_GPU, num_rois, (y1, x1, y2, x2)]，我们将进行最后的目标检测精修部分。
# Detections
# output is [batch, num_detections, (y1, x1, y2, x2, class_id, score)] in
# normalized coordinates
detections = DetectionLayer(config, name="mrcnn_detection")(
[rpn_rois, mrcnn_class, mrcnn_bbox, input_image_meta])
1、原始图片resize参数"window"
注意到我们的输入中一个input_image_meta项，它记录了每一张图片的原始信息，[batch, n]维矩阵，n是固定的，其生成与config.py文件中
# Image meta data length
# See compose_image_meta() for details
self.IMAGE_META_SIZE = 1 + 3 + 3 + 4 + 1 + self.NUM_CLASSES
其信息在未来的（如果有的话）图像预处理中会介绍，本节使用了其中记录的原图大小信息和对应图片的"window"信息。图片大小信息为3个整数，对应输入图片（即已经预处理之后的图片）的长宽和深度，"window"信息包含4个整数，其含义为(top_pad, left_pad, h + top_pad, w + left_pad)，和重置图片大小的处理有关，下面代码见utils.py的resize_image函数，
if mode == "square":
# Get new height and width
h, w = image.shape[:2]
top_pad = (max_dim - h) // 2
bottom_pad = max_dim - h - top_pad
left_pad = (max_dim - w) // 2
right_pad = max_dim - w - left_pad
padding = [(top_pad, bottom_pad), (left_pad, right_pad), (0, 0)]
image = np.pad(image, padding, mode='constant', constant_values=0)
window = (top_pad, left_pad, h + top_pad, w + left_pad)
即我们将深蓝色的原图（不要求w等于h）通过填充的方式扩展为浅灰色的大图用于feed网络，"window"记录了以新图左上角为原点建立坐标系，原图的左上角点和右下角点的坐标，由于坐标系选取的是像素坐标，"window"记录的就是原始图片的大小，其蕴含了输入图片中真正有意义的位置信息。
2、从"window"还原原始图片大小
有一点注意，假如top_pad=5，也就是我们在图像顶部填充了5行，实际上0、1、2、3、4为非图像区域，所以我们从第5行开始是图像；假设图像有3行（很极端），即5、6、7行为图像，但是:
top_pad+h=5+3=8
即[top_pad:top_pad+h-1]行为真实图片，列同理。
另外，用于解析image_meta结构的函数如下：
def parse_image_meta_graph(meta):
"""Parses a tensor that contains image attributes to its components.
See compose_image_meta() for more details.
meta: [batch, meta length] where meta length depends on NUM_CLASSES
Returns a dict of the parsed tensors.
"""
image_id = meta[:, 0]
original_image_shape = meta[:, 1:4]
image_shape = meta[:, 4:7]
window = meta[:, 7:11] # (y1, x1, y2, x2) window of image in in pixels
scale = meta[:, 11]
active_class_ids = meta[:, 12:]
return {
"image_id": image_id,
"original_image_shape": original_image_shape,
"image_shape": image_shape,
"window": window,
"scale": scale,
"active_class_ids": active_class_ids,
}
二、源码讲解
首先接收参数，初始化网络，
class DetectionLayer(KE.Layer):
"""Takes classified proposal boxes and their bounding box deltas and
returns the final detection boxes.
Returns:
[batch, num_detections, (y1, x1, y2, x2, class_id, class_score)] where
coordinates are normalized.
"""
def __init__(self, config=None, **kwargs):
super(DetectionLayer, self).__init__(**kwargs)
self.config = config
def call(self, inputs):
rois = inputs[0] # [batch, num_rois, (y1, x1, y2, x2)]
mrcnn_class = inputs[1] # [batch, num_rois, NUM_CLASSES]
mrcnn_bbox = inputs[2] # [batch, num_rois, NUM_CLASSES, (dy, dx, log(dh), log(dw))]
image_meta = inputs[3]
1、原始图片尺寸获取
然后获取"window"参数即原始图片尺寸，然后获取其相对于输入图片的image_shape即[w, h, channels]的尺寸，
# Get windows of images in normalized coordinates. Windows are the area
# in the image that excludes the padding.
# Use the shape of the first image in the batch to normalize the window
# because we know that all images get resized to the same size.
m = parse_image_meta_graph(image_meta)
image_shape = m['image_shape'][0]
window = norm_boxes_graph(m['window'], image_shape[:2]) # (y1, x1, y2, x2)
上面第5行调用函数如下（本文第一节中已经贴了），用于解析并获取输入图片的shape和原始图片的shape（即"window"）。第7行函数如下：
def norm_boxes_graph(boxes, shape):
"""Converts boxes from pixel coordinates to normalized coordinates.
boxes: [..., (y1, x1, y2, x2)] in pixel coordinates
shape: [..., (height, width)] in pixels
Note: In pixel coordinates (y2, x2) is outside the box. But in normalized
coordinates it's inside the box.
Returns:
[..., (y1, x1, y2, x2)] in normalized coordinates
"""
h, w = tf.split(tf.cast(shape, tf.float32), 2)
scale = tf.concat([h, w, h, w], axis=-1) - tf.constant(1.0)
shift = tf.constant([0., 0., 1., 1.])
return tf.divide(boxes - shift, scale)
我们经过"window"获取了原始图片相对输入图片的坐标（像素空间），然后除以输入图片的宽高，得到了原始图片相对于输入图片的normalized坐标，分布于[0,1]区间上。
事实上由于anchors生成的4个坐标值均位于[0,1]，在网络中所有的坐标都是位于[0,1]的，原始图片信息是新的被引入的量，不可或缺的需要被处理到正则空间。
对于每一张图片，我们有：
每个推荐区域的坐标
每个推荐区域的粗分类情况
每个推荐区域的坐标粗修
图片中真正有意义的位置坐标
下面我们基于这些信息，进行精提。
2、分类、回归信息精炼
# Run detection refinement graph on each item in the batch
detections_batch = utils.batch_slice(
[rois, mrcnn_class, mrcnn_bbox, window],
lambda x, y, w, z: refine_detections_graph(x, y, w, z, self.config),
注意，下面调用的函数，每次处理的是单张图片。
逻辑流程如下：
a 获取每个推荐区域得分最高的class的得分
b 获取每个推荐区域经过粗修后的坐标和"window"交集的坐标
c 剔除掉最高得分为背景的推荐区域
d 剔除掉最高得分达不到阈值的推荐区域
e 对属于同一类别的候选框进行非极大值抑制
f 对非极大值抑制后的框索引：剔除-1占位符，获取top k（100）
最后返回每个框(y1, x1, y2, x2, class_id, score)信息
step1
调用函数前半部分如下，
def refine_detections_graph(rois, probs, deltas, window, config):
"""Refine classified proposals and filter overlaps and return final
detections.
Inputs:
rois: [N, (y1, x1, y2, x2)] in normalized coordinates
probs: [N, num_classes]. Class probabilities.
deltas: [N, num_classes, (dy, dx, log(dh), log(dw))]. Class-specific
bounding box deltas.
window: (y1, x1, y2, x2) in normalized coordinates. The part of the image
that contains the image excluding the padding.
Returns detections shaped: [num_detections, (y1, x1, y2, x2, class_id, score)] where
coordinates are normalized.
"""
# Class IDs per ROI
class_ids = tf.argmax(probs, axis=1, output_type=tf.int32) # [N]，每张图片最高得分类
# Class probability of the top class of each ROI
indices = tf.stack([tf.range(probs.shape[0]), class_ids], axis=1) # [N, (图片序号, 最高class序号)]
class_scores = tf.gather_nd(probs, indices) # [N], 每张图片最高得分类得分值
# Class-specific bounding box deltas
deltas_specific = tf.gather_nd(deltas, indices) # [N, 4]
# Apply bounding box deltas
# Shape: [boxes, (y1, x1, y2, x2)] in normalized coordinates
refined_rois = apply_box_deltas_graph(
rois, deltas_specific * config.BBOX_STD_DEV) # [N, 4]
# Clip boxes to image window
refined_rois = clip_boxes_graph(refined_rois, window)
# TODO: Filter out boxes with zero area
# Filter out background boxes
# class_ids: N, where(class_ids > 0): [M, 1] 即where会升维
keep = tf.where(class_ids > 0)[:, 0]
# Filter out low confidence boxes
if config.DETECTION_MIN_CONFIDENCE: # 0.7
conf_keep = tf.where(class_scores >= config.DETECTION_MIN_CONFIDENCE)[:, 0]
# 求交集，返回稀疏Tensor，要求a、b除最后一维外维度相同，最后一维的各个子列分别求交集
# a = np.array([[{1, 2}, {3}], [{4}, {5, 6}]])
# b = np.array([[{1} , {}] , [{4}, {5, 6, 7, 8}]])
# res = np.array([[{1} , {}] , [{4}, {5, 6}]])
keep = tf.sets.set_intersection(tf.expand_dims(keep, 0),
tf.expand_dims(conf_keep, 0))
keep = tf.sparse_tensor_to_dense(keep)[0]
# Apply per-class NMS
# 1. Prepare variables
pre_nms_class_ids = tf.gather(class_ids, keep) # [n]
pre_nms_scores = tf.gather(class_scores, keep) # [n]
pre_nms_rois = tf.gather(refined_rois, keep) # [n, 4]
unique_pre_nms_class_ids = tf.unique(pre_nms_class_ids)[0] # 去重后class类别
'''
# tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
y, idx = unique(x)
y ==> [1, 2, 4, 7, 8]
idx ==> [0, 0, 1, 2, 2, 2, 3, 4, 4]
'''
这一部分代码主要对于当前的信息进行整理为精炼做准备，流程很清晰：
a 获取每个推荐区域得分最高的class的得分
b 获取每个推荐区域经过粗修后的坐标和"window"交集的坐标
c 剔除掉最高得分为背景的推荐区域
d 剔除掉最高得分达不到阈值的推荐区域
此时使用张量keep保存符合条件的推荐区域的index，即一个一维数组，每个值为一个框的序号，后面会继续对这个keep中的序号做进一步的筛选。
step2
e 对属于同一类别的候选框进行非极大值抑制。
注意下面的内嵌函数，包含keep（step1中保留的框索引）、pre_nms_class_ids（step1中保留的框类别）、pre_nms_scores（step1中保留的框得分）几个外部变量，
def nms_keep_map(class_id):
"""Apply Non-Maximum Suppression on ROIs of the given class."""
# 接受了外部变量pre_nms_class_ids、keep
# Indices of ROIs of the given class
# class_id表示当前NMS的目标类的数字，pre_nms_class_ids为全部的疑似目标类
ixs = tf.where(tf.equal(pre_nms_class_ids, class_id))[:, 0]
# Apply NMS
class_keep = tf.image.non_max_suppression(
tf.gather(pre_nms_rois, ixs), # 当前class的全部推荐区坐标
tf.gather(pre_nms_scores, ixs), # 当前class的全部推荐区得分
max_output_size=config.DETECTION_MAX_INSTANCES, # 100
iou_threshold=config.DETECTION_NMS_THRESHOLD) # 0.3
# Map indices
# class_keep是对ixs的索引，ixs是对keep的索引
class_keep = tf.gather(keep, tf.gather(ixs, class_keep)) # 由索引获取索引
# Pad with -1 so returned tensors have the same shape
gap = config.DETECTION_MAX_INSTANCES - tf.shape(class_keep)[0]
class_keep = tf.pad(class_keep, [(0, gap)],
mode='CONSTANT', constant_values=-1)
# Set shape so map_fn() can infer result shape
class_keep.set_shape([config.DETECTION_MAX_INSTANCES])
# 返回长度必须固定，否则tf.map_fn不能正常运行
return class_keep
# 2. Map over class IDs
nms_keep = tf.map_fn(nms_keep_map, unique_pre_nms_class_ids,
dtype=tf.int64) # [?, 默认100]：类别顺序，每个类别中的框索引
本步骤输出nms_keep，[?, 100]格式，？表示该张图片中保留的类别数（不是实例数注意）。
step3
f 对非极大值抑制后的框索引：剔除-1占位符，获取top k（100），返回每个框(y1, x1, y2, x2, class_id, score)信息。
# 3. Merge results into one list, and remove -1 padding
nms_keep = tf.reshape(nms_keep, [-1]) # 全部框索引
nms_keep = tf.gather(nms_keep, tf.where(nms_keep > -1)[:, 0]) # 剔除-1索引
# 4. Compute intersection between keep and nms_keep
# nms_keep本身就是从keep中截取的，本步感觉冗余
keep = tf.sets.set_intersection(tf.expand_dims(keep, 0),
tf.expand_dims(nms_keep, 0))
keep = tf.sparse_tensor_to_dense(keep)[0]
# Keep top detections
roi_count = config.DETECTION_MAX_INSTANCES
class_scores_keep = tf.gather(class_scores, keep) # 获取得分
num_keep = tf.minimum(tf.shape(class_scores_keep)[0], roi_count)
top_ids = tf.nn.top_k(class_scores_keep, k=num_keep, sorted=True)[1]
keep = tf.gather(keep, top_ids) # 由索引获取索引
# Arrange output as [N, (y1, x1, y2, x2, class_id, score)]
# Coordinates are normalized.
detections = tf.concat([
tf.gather(refined_rois, keep), # 索引坐标[?, 4]
tf.to_float(tf.gather(class_ids, keep))[..., tf.newaxis], # 索引class，添加维[?, 1]
tf.gather(class_scores, keep)[..., tf.newaxis] # 索引的分，添加维[?, 1]
], axis=1)
# 如果 detections < DETECTION_MAX_INSTANCES，填充0
gap = config.DETECTION_MAX_INSTANCES - tf.shape(detections)[0]
detections = tf.pad(detections, [(0, gap), (0, 0)], "CONSTANT")
return detections
至此，我们得到了可以用于输出的目标检测结果，下一步就是Mask信息生成。
