WebRTC中视频包发送流程

一、总体发送流程

参考代码版本：m96  

主要记录视频编码、帧处理成RTP包、Pacer发送、底层网络传输流程。

下文主要摘取出了主要流程，很多处理包的细节暂时没有展开。

1、涉及模块

 图片来源

 

2、主要流程调用堆栈

视频编码输出到视频数据包进发送队列

PacingController::EnqueuePacket 
PacedSender::EnqueuePackets
RTPSender::EnqueuePackets
RTPSenderVideo::LogAndSendToNetwork
RTPSenderVideo::SendVideo
RTPSenderVideo::SendEncodedImage
RtpVideoSender::OnEncodedImage
VideoSendStreamImpl::OnEncodedImage
VideoStreamEncoder::OnEncodedImage
VideoStreamEncoder::EncodeVideoFrame
VideoStreamEncoder::MaybeEncodeVideoFrame
VideoStreamEncoder::OnEncoderSettingsChanged()

Pacer发送

// pacing的线程发包调用堆栈
WebRtcVideoChannel::SendRtp  // 
RtpSenderEgress::SendPacketToNetwork
RtpSenderEgress::SendPacket
ModuleRtpRtcpImpl2::TrySendPacket
PacketRouter::SendPacket
PacingController::ProcessPackets()
PacedSender::Process()
ProcessThreadImpl::Process()

底层网络传输：

RtpTransport::SendPacket
SrtpTransport::SendRtcpPacket
BaseChannel::SendPacket
BaseChannel::SendRtcp
MediaChannel::DoSendPacket
MediaChannel::SendRtcp

 

二、具体发送函数

 1、编码输出

VideoStreamEncoder：原始视频帧---》编码bit 流

// VideoStreamEncoder represent a video encoder that accepts raw video frames as
// input and produces an encoded bit stream.
// Usage:
//  Instantiate.
//  Call SetSink.
//  Call SetSource.
//  Call ConfigureEncoder with the codec settings.
//  Call Stop() when done.
class VideoStreamEncoder : public VideoStreamEncoderInterface,
                           private EncodedImageCallback,
                           public VideoSourceRestrictionsListener {}； 

 

2、处理RTP包

RtpVideoSender

// RtpVideoSender routes outgoing data to the correct sending RTP module, based
// on the simulcast layer in RTPVideoHeader.
class RtpVideoSender : public RtpVideoSenderInterface,
                       public VCMProtectionCallback,
                       public StreamFeedbackObserver 
{

　　// Rtp modules are assumed to be sorted in simulcast index order.
　　const std::vector<webrtc_internal_rtp_video_sender::RtpStreamSender> rtp_streams_;

    RtpTransportControllerSendInterface* const transport_;

};

 RtpVideoSender中调用rtp_streams_[stream_index].sender_video->SendEncodedImage()方法将编码好的视频帧传给RtpSenderVideo。

namespace webrtc_internal_rtp_video_sender {
// RTP state for a single simulcast stream. Internal to the implementation of
// RtpVideoSender.
struct RtpStreamSender {

  // Note: Needs pointer stability.
  std::unique_ptr<ModuleRtpRtcpImpl2> rtp_rtcp; // rtp/rtcp组包、接收、发送
  std::unique_ptr<RTPSenderVideo> sender_video;  // pacer发送
  std::unique_ptr<VideoFecGenerator> fec_generator; // fec
};

}  // namespace webrtc_internal_rtp_video_sender

RTPSenderVideo::SendVideo中主要将视频帧的video_header、payload信息构造RTP包。

主要功能：

1、根据video_header来设置rtp包的头部扩展；

2、使用Packetizer打包rtp包；

3、有fec或red的话，做相应的处理；

4、调用RTPSenderVideo::LogAndSendToNetwork，将rtp包传给pacer发送。

rtp_sender_->EnqueuePackets(std::move(packets));---》paced_sender_->EnqueuePackets(std::move(packets));

class RTPSenderVideo 
{
　　RTPSender* const rtp_sender_;
};
class RTPSender
{
　　RtpPacketSender* const paced_sender_;
};

 

3、Pacer发送

PacedSender:

class PacedSender : public Module,
                    public RtpPacketPacer,
                    public RtpPacketSender {};

void PacedSender::EnqueuePackets(
    std::vector<std::unique_ptr<RtpPacketToSend>> packets) {
  {
    for (auto& packet : packets) {
      pacing_controller_.EnqueuePacket(std::move(packet));
    }
  }
  MaybeWakupProcessThread(); // 唤醒处理线程调用注册的process()去处理发送队列
}

 线程wakeup函数最后调用PacedSender::Process()。

void PacedSender::Process() {
  MutexLock lock(&mutex_);
  pacing_controller_.ProcessPackets();
}

// This class implements a leaky-bucket packet pacing algorithm. It handles the
// logic of determining which packets to send when, but the actual timing of
// the processing is done externally (e.g. PacedSender). Furthermore, the
// forwarding of packets when they are ready to be sent is also handled
// externally, via the PacedSendingController::PacketSender interface.
//
class PacingController 
{

  // Periodic mode uses the IntervalBudget class for tracking bitrate
  // budgets, and expected ProcessPackets() to be called a fixed rate,
  // e.g. every 5ms as implemented by PacedSender.
  // Dynamic mode allows for arbitrary time delta between calls to
  // ProcessPackets.
  enum class ProcessMode { kPeriodic, kDynamic };

  class PacketSender {
   public:
    virtual ~PacketSender() = default;
    virtual void SendPacket(std::unique_ptr<RtpPacketToSend> packet,
                            const PacedPacketInfo& cluster_info) = 0;
    // Should be called after each call to SendPacket().
    virtual std::vector<std::unique_ptr<RtpPacketToSend>> FetchFec() = 0;
    virtual std::vector<std::unique_ptr<RtpPacketToSend>> GeneratePadding(
        DataSize size) = 0;
  };

    PacketSender* const packet_sender_;   // 发包的模块
}

PacingController::ProcessPackets()中使用PacketSender发送RTP包，调用PacketRouter::SendPacket。

PacingController平滑发送的原理：（这里使用的是周期（5ms）发送，最新的使用dynamic发送的pacer是基于debt机制）

　　1、使用budge控制发送码率；

　　2、UpdateBudgetWithElapsedTime()根据时间流逝更新budge；

　　3、根据发送队列大小和包在队列存放的时间计算一个发送码率。DataRate min_rate_needed = queue_size_data / avg_time_left;

　　4、获取BitrateProber探测到的带宽的结果。recommended_probe_size = prober_.RecommendedMinProbeSize();

　　5、在一个while循环中发送数据包。packet_sender_->SendPacket(std::move(rtp_packet), pacing_info)。

 

// PacketRouter keeps track of rtp send modules to support the pacer.
// In addition, it handles feedback messages, which are sent on a send
// module if possible (sender report), otherwise on receive module
// (receiver report). For the latter case, we also keep track of the
// receive modules.
class PacketRouter : public PacingController::PacketSender 
{};

PacketRouter::SendPacket中主要

　　1、设置头部扩展 packet->SetExtension<TransportSequenceNumber>((++transport_seq_) & 0xFFFF);

　　2、发包

　　　　RtpRtcpInterface* rtp_module = kv->second;
　　　　if (!rtp_module->TrySendPacket(packet.get(), cluster_info))

　　3、暂存fec包，看是否需要发送；

　　　　pending_fec_packets_.push_back(std::move(packet))。

 

ModuleRtpRtcpImpl2::TrySendPacket中是调用rtp_sender_->packet_sender.SendPacket(packet, pacing_info);

class ModuleRtpRtcpImpl2 final : public RtpRtcpInterface,
                                 public RTCPReceiver::ModuleRtpRtcp 
{
  //
  bool TrySendPacket(RtpPacketToSend* packet,
                     const PacedPacketInfo& pacing_info) override;
  // Called when we receive an RTCP packet.
  void IncomingRtcpPacket(const uint8_t* incoming_packet,
                          size_t incoming_packet_length) override;
  
  struct RtpSenderContext {
    explicit RtpSenderContext(const RtpRtcpInterface::Configuration& config);
    // Storage of packets, for retransmissions and padding, if applicable.
    RtpPacketHistory packet_history;
    SequenceChecker sequencing_checker;
    // Handles sequence number assignment and padding timestamp generation.
    PacketSequencer sequencer RTC_GUARDED_BY(sequencing_checker);
    // Handles final time timestamping/stats/etc and handover to Transport.
    RtpSenderEgress packet_sender;
    // If no paced sender configured, this class will be used to pass packets
    // from `packet_generator_` to `packet_sender_`.
    RtpSenderEgress::NonPacedPacketSender non_paced_sender;
    // Handles creation of RTP packets to be sent.
    RTPSender packet_generator;
  };
  //
  // 用于发送接收RTP和RTCP包
  std::unique_ptr<RtpSenderContext> rtp_sender_;
  RTCPSender rtcp_sender_;
  RTCPReceiver rtcp_receiver_; 
};

 

RTPSenderExgress

bool RtpSenderEgress::SendPacketToNetwork(const RtpPacketToSend& packet,
                                          const PacketOptions& options,
                                          const PacedPacketInfo& pacing_info) {
    bytes_sent = transport_->SendRtp(packet.data(), packet.size(), options) // 调用WebRtcVideoChannel::SendRtp
}

bool WebRtcVideoChannel::SendRtp(const uint8_t* data,
                                 size_t len,
                                 const webrtc::PacketOptions& options) {
  MediaChannel::SendRtp(data, len, options);
  return true;
}

 

4、底层网络发送

底层网络传输代用堆栈：

RtpTransport::SendPacket
SrtpTransport::SendRtcpPacket
BaseChannel::SendPacket
BaseChannel::SendRtcp
MediaChannel::DoSendPacket
MediaChannel::SendRtcp

 

RTPTransport::SendPacket

bool RtpTransport::SendPacket(bool rtcp,
                              rtc::CopyOnWriteBuffer* packet,
                              const rtc::PacketOptions& options,
                              int flags) {
  rtc::PacketTransportInternal* transport = rtcp && !rtcp_mux_enabled_
                                                ? rtcp_packet_transport_
                                                : rtp_packet_transport_;
  // 发送的函数
  int ret = transport->SendPacket(packet->cdata<char>(), packet->size(),
                                  options, flags); 
}

 三、影响视频发送的重要参数

1、PacingController::media_budget_。This is the media budget, keeping track of how many bits of media we can pace out during the current interval.

2、RoundRobinPacketQueue PacingController::packet_queue_。队列中的数据包。

3、recommended_probe_size = prober_.RecommendedMinProbeSize(); 探测的带宽大小。

todo：需要再理解RTT的探测、重传的周期等内容。

 

参考文献：

视频发送流程 （自己理完代码，发现这一篇写得很详细）