Opus Packet Loss Concealment

[rvadhavk]

From the libopus documentation, it looks like the way to handle missing packets and have the decoder output packet loss concealment is to pass it an empty/null buffer.

But when creating an EncodedAudioChunk with a zero-length buffer and passing it to the decoder, in Chrome I get an error of "DOMException: Null or empty decoder buffer." The check in Chrome is here.

Is this the expected behavior, a Chrome implementation issue, or is this behavior something that should be clarified in the standard?

[bdrtc]

the same issue exist for decode opus inband fec when packet loss, chrome does not impl this and no describe for such scenario in standard。

[dalecurtis]

@sandersdan @tguilbert-google

[sandersdan]

[Edit: replacing comment]

This doesn't appear in the spec, not sure why Chrome is enforcing it.

I filed https://bugs.chromium.org/p/chromium/issues/detail?id=1363687

[sandersdan]

Reopening as a discussion topic: should WebCodecs be exposing loss concealment, and if so is an empty buffer the right API?

[dalecurtis]

If this is how the opus library has standardized signaling and it's what developers / RTC implementations are most familiar with, we probably should. I expect the null packet does something better than what would happen if we required developers to call flush() and skip these packets?

[aboba]

Agree with @dalecurtis. The null packet will indeed do something better than calling flush(), whether that is concealment or potentially enabling recovery when FEC is enabled.

Note: Opus FEC is distance 1 so it cannot handle burst loss. So until Opus 1.5 DRED is available, applications will typically need to support redundant audio (RED), which is external to WebCodecs. For a discussion of RED, see: https://webrtchacks.com/red-improving-audio-quality-with-redundancy/

[sandersdan]

I've been testing in Chrome and it does not appear to be possible to activate this feature in libopus via ffmpeg, as ffmpeg interprets an empty packet as EOS. This likely means that the feature will not be implemented in Chrome in the short term.

[dalecurtis]

Editor's call: Allow null packets for opus. Editors were not aware of any other codecs which have this mechanism, so not worth a new encoded audio chunk type or similar. Probably worth a note in the registry that this is how FEC works for opus.

[sandersdan]

FYI: There is a proposed FFmpeg patch (appears to be abandoned) to implement this API; that patch worked by turning PTS discontinuities into FEC / error concealment.

[aboba]

@dalecurtis Should this be labeled "Extension"?