关于Gstreamer出现“Could not send sticky events”的机制探究
问题描述
一个双目视觉的插件，之前输入分辨率540*960，输出分辨率540*960，现在需要修改输出分辨率为1080*1920，结果修改了event function之后，前面的插件报出WARNING Could not send sticky events还有result not-negotiated, mark pending events。
之前并不是很懂gstreamer的event和query机制，经过两天时间扒gstreamer源码，对gstreamer机制有了更深的了解，来记录一下。
Gstreamer event事件定义：
这里直接上event事件的源码定义：
typedef enum {
GST_EVENT_TYPE_UPSTREAM = 1 << 0,
GST_EVENT_TYPE_DOWNSTREAM = 1 << 1,
GST_EVENT_TYPE_SERIALIZED = 1 << 2,
GST_EVENT_TYPE_STICKY = 1 << 3,
GST_EVENT_TYPE_STICKY_MULTI = 1 << 4
} GstEventTypeFlags;
#define GST_EVENT_NUM_SHIFT (8)
#define GST_EVENT_MAKE_TYPE(num,flags) \
(((num) << GST_EVENT_NUM_SHIFT) | (flags))
#define FLAG(name) GST_EVENT_TYPE_##name
typedef enum {
GST_EVENT_UNKNOWN = GST_EVENT_MAKE_TYPE (0, 0),
/* bidirectional events */
GST_EVENT_FLUSH_START = GST_EVENT_MAKE_TYPE (10, FLAG(BOTH)),
GST_EVENT_FLUSH_STOP = GST_EVENT_MAKE_TYPE (20, FLAG(BOTH) | FLAG(SERIALIZED)),
/* downstream serialized events */
GST_EVENT_STREAM_START = GST_EVENT_MAKE_TYPE (40, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_CAPS = GST_EVENT_MAKE_TYPE (50, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_SEGMENT = GST_EVENT_MAKE_TYPE (70, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_TAG = GST_EVENT_MAKE_TYPE (80, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_BUFFERSIZE = GST_EVENT_MAKE_TYPE (90, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_SINK_MESSAGE = GST_EVENT_MAKE_TYPE (100, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_EOS = GST_EVENT_MAKE_TYPE (110, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY)),
GST_EVENT_TOC = GST_EVENT_MAKE_TYPE (120, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_PROTECTION = GST_EVENT_MAKE_TYPE (130, FLAG (DOWNSTREAM) | FLAG (SERIALIZED) | FLAG (STICKY) | FLAG (STICKY_MULTI)),
/* non-sticky downstream serialized */
GST_EVENT_SEGMENT_DONE = GST_EVENT_MAKE_TYPE (150, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
GST_EVENT_GAP = GST_EVENT_MAKE_TYPE (160, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
/* upstream events */
GST_EVENT_QOS = GST_EVENT_MAKE_TYPE (190, FLAG(UPSTREAM)),
GST_EVENT_SEEK = GST_EVENT_MAKE_TYPE (200, FLAG(UPSTREAM)),
GST_EVENT_NAVIGATION = GST_EVENT_MAKE_TYPE (210, FLAG(UPSTREAM)),
GST_EVENT_LATENCY = GST_EVENT_MAKE_TYPE (220, FLAG(UPSTREAM)),
GST_EVENT_STEP = GST_EVENT_MAKE_TYPE (230, FLAG(UPSTREAM)),
GST_EVENT_RECONFIGURE = GST_EVENT_MAKE_TYPE (240, FLAG(UPSTREAM)),
GST_EVENT_TOC_SELECT = GST_EVENT_MAKE_TYPE (250, FLAG(UPSTREAM)),
/* custom events start here */
GST_EVENT_CUSTOM_UPSTREAM = GST_EVENT_MAKE_TYPE (270, FLAG(UPSTREAM)),
GST_EVENT_CUSTOM_DOWNSTREAM = GST_EVENT_MAKE_TYPE (280, FLAG(DOWNSTREAM) | FLAG(SERIALIZED)),
GST_EVENT_CUSTOM_DOWNSTREAM_OOB = GST_EVENT_MAKE_TYPE (290, FLAG(DOWNSTREAM)),
GST_EVENT_CUSTOM_DOWNSTREAM_STICKY = GST_EVENT_MAKE_TYPE (300, FLAG(DOWNSTREAM) | FLAG(SERIALIZED) | FLAG(STICKY) | FLAG(STICKY_MULTI)),
GST_EVENT_CUSTOM_BOTH = GST_EVENT_MAKE_TYPE (310, FLAG(BOTH) | FLAG(SERIALIZED)),
GST_EVENT_CUSTOM_BOTH_OOB = GST_EVENT_MAKE_TYPE (320, FLAG(BOTH))
} GstEventType;
Gstreamer是如何设计宏的，从这里其实很清楚。
GST_EVENT_MAKE_TYPE 宏其实只做了一件很简单的事情，将第一个数字参数左移8位，空出来的低8位用于存储该事件拥有什么性质，这一点在GstEventTypeFlags枚举变量有说明，相应位为1或0代表有无该性质。这种C语言设计宏的技巧在我们自己写C/C++程序的时候也可以用得到。
另外，可以看到，很多常见的事件都是sticky事件，如CAPS和EOS，所以Gstreamer出现“Could not send sticky events”话首先要判断是什么事件。
gst_pad_peer_query 函数
gboolean
gst_pad_peer_query (GstPad * pad, GstQuery * query)
{
[...]
serialized = GST_QUERY_IS_SERIALIZED (query);
GST_OBJECT_LOCK (pad);
if (GST_PAD_IS_SRC (pad) && serialized) {
/* all serialized queries on the srcpad trigger push of
* sticky events */
if (check_sticky (pad, NULL) != GST_FLOW_OK)
goto sticky_failed;
}
[...]
peerpad = GST_PAD_PEER (pad);
[...]
res = gst_pad_query (peerpad, query);
[...]
return res;
/* ERRORS */
[...]
sticky_failed:
{
GST_WARNING_OBJECT (pad, "could not send sticky events");
GST_OBJECT_UNLOCK (pad);
return FALSE;
}
[...]
}
}
无关紧要的部分做了删减，gst_pad_peer_query 是用来给pad的 peerpad（即与该pad连接成功的且绑定在一起的pad）发送query用的，实际上就是给peerpad调用gst_pad_query函数。
"could not send sticky events"则是由于check_sticky函数失败造成的，那么这个函数是干什么的呢？
在这里，其内部主要是判断一下当前pad有没有滞留的事件，如果有，那么就把它们发送给peerpad并清空。
check_sticky函数
/* check sticky events and push them when needed. should be called
* with pad LOCK */
static inline GstFlowReturn
check_sticky (GstPad * pad, GstEvent * event)
{
PushStickyData data = { GST_FLOW_OK, FALSE, event };
if (G_UNLIKELY (GST_PAD_HAS_PENDING_EVENTS (pad))) {
GST_OBJECT_FLAG_UNSET (pad, GST_PAD_FLAG_PENDING_EVENTS);
GST_DEBUG_OBJECT (pad, "pushing all sticky events");
events_foreach (pad, push_sticky, &data);
/* If there's an EOS event we must push it downstream
* even if sending a previous sticky event failed.
* Otherwise the pipeline might wait forever for EOS.
*
* Only do this if pushing another event than the EOS
* event failed.
*/
if (data.ret != GST_FLOW_OK && !data.was_eos) {
PadEvent *ev = find_event_by_type (pad, GST_EVENT_EOS, 0);
if (ev && !ev->received) {
data.ret = gst_pad_push_event_unchecked (pad, gst_event_ref (ev->event),
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM);
/* the event could have been dropped. Because this can only
* happen if the user asked for it, it's not an error */
if (data.ret == GST_FLOW_CUSTOM_SUCCESS)
data.ret = GST_FLOW_OK;
}
}
}
return data.ret;
}
check_sticky源码很短，简单可以分成两部分：
- 通过
events_foreach来给未处理的事件迭代调用push_sticky函数 - 如果要处理的事件是EOS，哪怕它push失败了，也要传给下游，不然流水线结束不了。
GST_PAD_HAS_PENDING_EVENTS 宏从字面意思上看，是判断是不是有未处理的事件。上一个函数传的参数中，data结构体里的event其实就是NULL。
data.ret则是造成问题的原因，而这个ret只是被push_sticky修改了，events_foreach并未做改动，只是单纯地把pad未处理的事件依次送到push_sticky的第二个参数（见下文），所以问题出现在push_sticky函数
push_sticky函数
static gboolean
push_sticky (GstPad * pad, PadEvent * ev, gpointer user_data)
{
PushStickyData *data = user_data;
GstEvent *event = ev->event;
if (ev->received) {
GST_DEBUG_OBJECT (pad, "event %s was already received",
GST_EVENT_TYPE_NAME (event));
return TRUE;
}
/* If we're called because of an sticky event, only forward
* events that would come before this new event and the
* event itself */
if (data->event && GST_EVENT_IS_STICKY (data->event) &&
GST_EVENT_TYPE (data->event) <= GST_EVENT_SEGMENT &&
GST_EVENT_TYPE (data->event) < GST_EVENT_TYPE (event)) {
data->ret = GST_FLOW_CUSTOM_SUCCESS_1;
} else {
data->ret = gst_pad_push_event_unchecked (pad, gst_event_ref (event),
GST_PAD_PROBE_TYPE_EVENT_DOWNSTREAM);
if (data->ret == GST_FLOW_CUSTOM_SUCCESS_1)
data->ret = GST_FLOW_OK;
}
switch (data->ret) {
case GST_FLOW_OK:
ev->received = TRUE;
GST_DEBUG_OBJECT (pad, "event %s marked received",
GST_EVENT_TYPE_NAME (event));
break;
case GST_FLOW_CUSTOM_SUCCESS:
/* we can't assume the event is received when it was dropped */
GST_DEBUG_OBJECT (pad, "event %s was dropped, mark pending",
GST_EVENT_TYPE_NAME (event));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
data->ret = GST_FLOW_OK;
break;
case GST_FLOW_CUSTOM_SUCCESS_1:
/* event was ignored and should be sent later */
GST_DEBUG_OBJECT (pad, "event %s was ignored, mark pending",
GST_EVENT_TYPE_NAME (event));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
data->ret = GST_FLOW_OK;
break;
case GST_FLOW_NOT_LINKED:
/* not linked is not a problem, we are sticky so the event will be
* rescheduled to be sent later on re-link, but only for non-EOS events */
GST_DEBUG_OBJECT (pad, "pad was not linked, mark pending");
if (GST_EVENT_TYPE (event) != GST_EVENT_EOS) {
data->ret = GST_FLOW_OK;
ev->received = TRUE;
}
break;
default:
GST_DEBUG_OBJECT (pad, "result %s, mark pending events",
gst_flow_get_name (data->ret));
GST_OBJECT_FLAG_SET (pad, GST_PAD_FLAG_PENDING_EVENTS);
break;
}
if (data->ret != GST_FLOW_OK && GST_EVENT_TYPE (event) == GST_EVENT_EOS)
data->was_eos = TRUE;
return data->ret == GST_FLOW_OK;
}
user_data则是之前的data指针。整个代码看似长，其实只调用了一个函数gst_pad_push_event_unchecked ，除了这个函数以外的代码不可能让ret值是not-negotiated的。
如果在gstreamer里开启debug level为DEBUG级的话，还可以看到result not-negotiated, mark pending events的字眼。这个 not-negotiated就是data->ret的值，也就是这个原因导致了“Could not send sticky events”错误。接下来进入gst_pad_push_event_unchecked 函数。
补充：关于GstFlowReturn枚举类型的定义，前述的not-negotiated其实就是GST_FLOW_NOT_NEGOTIATED = -4这一宏，所以下文debug的时候应该注意是什么地方让ret变量赋了个GST_FLOW_NOT_NEGOTIATED
typedef enum {
/* custom success starts here */
GST_FLOW_CUSTOM_SUCCESS_2 = 102,
GST_FLOW_CUSTOM_SUCCESS_1 = 101,
GST_FLOW_CUSTOM_SUCCESS = 100,
/* core predefined */
GST_FLOW_OK = 0,
/* expected failures */
GST_FLOW_NOT_LINKED = -1,
GST_FLOW_FLUSHING = -2,
/* error cases */
GST_FLOW_EOS = -3,
GST_FLOW_NOT_NEGOTIATED = -4,
GST_FLOW_ERROR = -5,
GST_FLOW_NOT_SUPPORTED = -6,
/* custom error starts here */
GST_FLOW_CUSTOM_ERROR = -100,
GST_FLOW_CUSTOM_ERROR_1 = -101,
GST_FLOW_CUSTOM_ERROR_2 = -102
} GstFlowReturn;
gst_pad_push_event_unchecked 函数
static GstFlowReturn
gst_pad_push_event_unchecked (GstPad * pad, GstEvent * event,
GstPadProbeType type)
{
GstFlowReturn ret;
GstPad *peerpad;
GstEventType event_type;
gint64 old_pad_offset = pad->offset;
/* pass the adjusted event on. We need to do this even if
* there is no peer pad because of the probes. */
event = apply_pad_offset (pad, event, GST_PAD_IS_SINK (pad));
[...]
/* the pad offset might've been changed by any of the probes above. It
* would've been taken into account when repushing any of the sticky events
* above but not for our current event here */
if (G_UNLIKELY (old_pad_offset != pad->offset)) {
event =
_apply_pad_offset (pad, event, GST_PAD_IS_SINK (pad),
pad->offset - old_pad_offset);
}
[...]
/* now check the peer pad */
peerpad = GST_PAD_PEER (pad);
[...]
ret = gst_pad_send_event_unchecked (peerpad, event, type);
[...]
return ret;
[...]
}
type参数不太重要，所以删去了一些相关代码。这里重点关注何时返回的GST_FLOW_NOT_NEGOTIATED 的，可以看出来，主要是gst_pad_send_event_unchecked 这个函数，接下来继续进
gst_pad_send_event_unchecked 函数
static GstFlowReturn
gst_pad_send_event_unchecked (GstPad * pad, GstEvent * event,
GstPadProbeType type)
{
GstFlowReturn ret;
GstEventType event_type;
gboolean serialized, need_unlock = FALSE, sticky;
GstPadEventFunction eventfunc;
GstPadEventFullFunction eventfullfunc = NULL;
GstObject *parent;
[...]
event = apply_pad_offset (pad, event, GST_PAD_IS_SRC (pad));
[...]
eventfullfunc = GST_PAD_EVENTFULLFUNC (pad);
eventfunc = GST_PAD_EVENTFUNC (pad);
if (G_UNLIKELY (eventfunc == NULL && eventfullfunc == NULL))
goto no_function;
[...]
if (eventfullfunc) {
ret = eventfullfunc (pad, parent, event);
} else if (eventfunc (pad, parent, event)) {
ret = GST_FLOW_OK;
} else {
/* something went wrong */
switch (event_type) {
case GST_EVENT_CAPS:
ret = GST_FLOW_NOT_NEGOTIATED;
break;
default:
ret = GST_FLOW_ERROR;
break;
}
}
[...]
return ret;
[...]
}
这里依然重点关注何时返回的GST_FLOW_NOT_NEGOTIATED 的，注意这两个宏GST_PAD_EVENTFUNC 和GST_PAD_EVENTFULLFUNC，另外，注意这时候pad指针指向的是下游的pad的了，也就是一开始的pad的peerpad。根据官方说明，这两个宏是负责调用下游pad的event函数，这就是我们写插件的时候那个gst_pad_set_event_function()和gst_pad_set_event_full_function()设置的event function。
/**
* GST_PAD_EVENTFUNC:
* @pad: a #GstPad
*
* Get the #GstPadEventFunction from the given @pad, which
* is the function that handles events on the pad. You can
* use this to set your own event handling function on a pad
* after you create it. If your element derives from a base
* class, use the base class's virtual functions instead.
*/
#define GST_PAD_EVENTFUNC(pad) (GST_PAD_CAST(pad)->eventfunc)
/**
* GST_PAD_EVENTFULLFUNC:
* @pad: a #GstPad
*
* Get the #GstPadEventFullFunction from the given @pad, which
* is the function that handles events on the pad. You can
* use this to set your own event handling function on a pad
* after you create it. If your element derives from a base
* class, use the base class's virtual functions instead.
*
* Since: 1.8
*/
#define GST_PAD_EVENTFULLFUNC(pad) (GST_PAD_CAST(pad)->ABI.abi.eventfullfunc)
那么ret是何时被置为GST_FLOW_NOT_NEGOTIATED 呢，当eventfullfunc 和eventfunc 没有返回True的时候，就会这样，所以问题归结到自己写的event function何时返回True，何时返回False了。
总结
其实整个过程走了一遍之后，后面再遇到这种问题就可以直接判断是peerpad的event function返回0所致。
顺带一提，通常在event function中会调用gst_pad_push_event函数，这个函数进去看的话，如果event是sticky的，又会调用check_sticky函数，这和上文形成了闭环，整个gstreamer流水线就是这样一级一级调用函数传递event事件的。
浙公网安备 33010602011771号