[question] Stream does not close if there is no input

[thegobot]

When the browser does not send RPT data (for the test, I closed all incoming udp from browser to server), the video-agent (ws-transcoder) continues to send the last frame to the ouput streams.

Explain the logic to me. If there are no input frames, is the last frame sent? If so, how to prevent this and close the stream?

[thegobot]

I found the reason

Last frame sent m_activeI420Buffer:

owt-server/source/core/owt_base/FrameProcesser.cpp

Line 327 in 5fc8bf2

i420Buffer = m_activeI420Buffer;

As far as I understand, sending frames on a timer is needed for accurate FPS?
Webrtc :: VideoEncoder has a maxFramerate parameter, but as I understand it, it doesn't work?

owt-server/source/core/owt_base/VCMFrameEncoder.cpp

Line 173 in 5fc8bf2

codecSettings.maxFramerate = frameRate;

If I set m_outFrameRate = 0 in FrameProcesser.cpp (FrameProcesser::init) then this solves my problem, if new data does not arrive, then the stream is closed. But then you cannot limit the FPS

Any solution?

[thegobot]

The problem is in webrtc-agent. The agent should generate an error when there is no data, but it does not. The agent will send out the delivery endlessly.

What do I see in the logs when I block the data flow from the browser

# Webrtc
log4j.logger.DtlsTransport=DEBUG
log4j.logger.LibNiceConnection=DEBUG
log4j.logger.NicerConnection=DEBUG
log4j.logger.OneToManyProcessor=DEBUG
log4j.logger.Resender=DEBUG
log4j.logger.SdpInfo=DEBUG
log4j.logger.SrtpChannel=DEBUG
log4j.logger.Stats=DEBUG
log4j.logger.WebRtcConnection=DEBUG
log4j.logger.IceConnection=DEBUG
log4j.logger.MediaStream=DEBUG
log4j.logger.StreamStats=DEBUG

iptables -t raw -A PREROUTING -s 165.13.148.252 -p udp -j DROP

DEBUG: MediaStream - id: 647648519884154100, role:publisher,  deliverFeedback unknownSSRC: 2306801664, localVideoSSRC: 1370879523, localAudioSSRC: 1755116946
INFO: StreamStats - {"SL0TL0":0,"SL0TL1":0,"SL0TL2":0,"SL0TL3":0,"SL1TL0":0,"SL1TL1":0,"SL1TL2":0,"SL1TL3":0,"SL2TL0":0,"SL2TL1":0,"SL2TL2":0,"SL2TL3":0,"SL3TL0":0,"SL3TL1":0,"SL3TL2":0,"SL3TL3":0,"audioBitrate":0,"audioEnabled":1,"audioFL":0,"audioJitter":0,"audioMuted":0,"audioNack":0,"audioPL":0,"audioRemb":0,"bwe":0,"isPublisher":1,"maxActiveSL":0,"maxActiveTL":0,"maxVideoBW":300000,"paddingBitrate":0,"rtxBitrate":0,"selectedSL":0,"selectedTL":0,"slideshow":0,"streamId":"647648519884154100","totalBitrate":0,"videoBitrate":0,"videoEnabled":1,"videoErizoRemb":0,"videoFL":0,"videoFir":0,"videoJitter":0,"videoKeyFrames":0,"videoMuted":0,"videoNack":0,"videoPL":0,"videoPli":0,"videoRemb":0}

DEBUG: owt.AVStreamOut - Request video key frame
DEBUG: owt.VCMFrameEncoder - (0x2789500)requestKeyFrame(0

DEBUG: MediaStream - id: 647648519884154100, role:publisher,  deliverFeedback unknownSSRC: 2306801664, localVideoSSRC: 1370879523, localAudioSSRC: 1755116946
DEBUG: owt.AVStreamOut - Request video key frame
DEBUG: owt.VCMFrameEncoder - (0x2789500)requestKeyFrame(0)

...
Endless repetition
...

The stream will be endless and will not stop!

[thegobot]

I found solution
In erizoerizo::MediaStream::deliverFeedback_() I added notifyMediaStreamEvent("deliver", "unknownSSRC");

int MediaStream::deliverFeedback_(std::shared_ptr<DataPacket> fb_packet) {
  RtcpHeader *chead = reinterpret_cast<RtcpHeader*>(fb_packet->data);
  uint32_t recvSSRC = chead->getSourceSSRC();
  if (chead->isREMB()) {
    for (uint8_t index = 0; index < chead->getREMBNumSSRC(); index++) {
      uint32_t ssrc = chead->getREMBFeedSSRC(index);
      if (isVideoSourceSSRC(ssrc)) {
        recvSSRC = ssrc;
        break;
      }
    }
  }
  if (isVideoSourceSSRC(recvSSRC)) {
    fb_packet->type = VIDEO_PACKET;
    sendPacketAsync(std::make_shared<DataPacket>(*fb_packet));
  } else if (isAudioSourceSSRC(recvSSRC)) {
    fb_packet->type = AUDIO_PACKET;
    sendPacketAsync(std::make_shared<DataPacket>(*fb_packet));
  } else {
    ELOG_ERROR("%s deliverFeedback unknownSSRC: %u, localVideoSSRC: %u, localAudioSSRC: %u",
                toLog(), recvSSRC, this->getVideoSourceSSRC(), this->getAudioSourceSSRC());

    notifyMediaStreamEvent("deliver", "unknownSSRC");

  }
  return fb_packet->length;
}

then, in

owt-server/source/agent/webrtc/connection.js

Line 115 in 5fc8bf2

mediaStream.onMediaStreamEvent((type, message) => {

mediaStream.onMediaStreamEvent((type, message) => {
  this._onMediaStreamEvent(type, message, mediaStream.id);

  log.debug('media_stream_event', type, message);

  if(message === "unknownSSRC"){
    this.removeMediaStream(mediaStream.id);
  }

});


ERROR: MediaStream - id: 383473589000484800, role:publisher,  deliverFeedback unknownSSRC: 2491351040, localVideoSSRC: 2527576173, localAudioSSRC: 2782675742
DEBUG: Connection - media_stream_event deliver unknownSSRC
DEBUG: Connection - removed mediaStreamId 383473589000484800, remaining size 0
DEBUG: Connection - message: _maybeSendAnswer sending event, type: offer, streamId: 383473589000484800
DEBUG: WrtcConnection - status_event { type: 'offer', sdp: '...',  
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error>
what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
INFO: NodeManager - Node webrtc-302ead3616b441812651@172.17.0.2_1 is closed on unexpected code: null
DEBUG: Conference - onSessionAborted
DEBUG: VideoNode - unpublish, stream_id: 235608613513137660
DEBUG: mcu.media.VideoTranscoder - CloseAll
DEBUG: mcu.media.VideoTranscoder - Closed all media in this Transcoder

i delete the stuck stream. After blocking with a firewall, after about 7 seconds. the stream is closed.. This solution works for me.

[starwarfan]

Hi, I see a boost exception in your logs. The mediaStream's destruction should be very careful.
We want to know that if you wait for 15 seconds or more, will the stream be deleted automatically in current code base?

[thegobot]

Yes, I don't use acceleration. The stream is not deleted. Never.

[thegobot]

After several events , I am trying to remove the stuck stream.

webrtc_agent/webrtc/connection.js

  _createMediaStream(id, options = {}, isPublisher = true) {
    log.debug(`message: _createMediaStream, connectionId: ${this.id}, ` +
              `mediaStreamId: ${id}, isPublisher: ${isPublisher}`);
    const mediaStream = new addon.MediaStream(this.threadPool, this.wrtc, id,
      options.label, this._getMediaConfiguration(this.mediaConfiguration), isPublisher);
    mediaStream.id = id;
    mediaStream.label = options.label;
    if (options.metadata) {
      // mediaStream.metadata = options.metadata;
      // mediaStream.setMetadata(JSON.stringify(options.metadata));
    }
    mediaStream.onMediaStreamEvent((type, message) => {
      this._onMediaStreamEvent(type, message, mediaStream.id);

      log.debug('media_stream_event', this.countUnknownSSRC, type, message);

      if(message === "unknownSSRC" && this.countUnknownSSRC > 10){

        this.removeMediaStream(mediaStream.id); //try remove stuck stream!
      }

      this.countUnknownSSRC++;

    });
    return mediaStream;
  }

This throws an exception

terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
  what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument

in erizo/WebRtcConnection.cpp

void WebRtcConnection::removeMediaStream(const std::string& stream_id) {
  asyncTask([stream_id] (std::shared_ptr<WebRtcConnection> connection) {
    boost::mutex::scoped_lock lock(connection->update_state_mutex_);
 }

...