Skip to content

[WebCodecs] Allow encoder to drop some frames in case of realtime mode #42961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

youennf
Copy link
Contributor

@youennf youennf commented Nov 6, 2023

No description provided.

@youennf youennf changed the title Allow encoder to drop some frames in case of realtime mode [WebCodecs] Allow encoder to drop some frames in case of realtime mode Nov 6, 2023
@padenot padenot assigned youennf and unassigned Djuffin Nov 6, 2023
@padenot padenot merged commit 579be63 into web-platform-tests:master Nov 6, 2023
@Djuffin
Copy link
Contributor

Djuffin commented Nov 6, 2023

I haven't found in the spec how exactly VideoEncoder is supposed to signal that the frame is dropped.
This PR assumes that the output callback is never called for dropped frames.
I think it's better to still call the output callback with empty data to make frame drop more transparent.

Was this previously discussed?

@padenot
Copy link
Contributor

padenot commented Nov 6, 2023

This is correct per spec.

4.1. Attempt to use [[codec implementation]] to encode the frameClone according to options.
[...] error handling -- this is not considered an error
4.5. Let encoded outputs be a list of encoded video data outputs emitted by [[codec implementation]].
4.6. If encoded outputs is not empty, queue a task to run the Output EncodedVideoChunks algorithm with encoded outputs.

Step 4.5, 4.6, the implementation has some output, it might have decided to have dropped some input (this is legal per https://w3c.github.io/webcodecs/#dom-latencymode-realtime), and if it has some output, it calls the callback.

We discussed this, e.g. in your message here: w3c/webcodecs#240 (comment) (and you had the same opinion, but also proposed the current specified behavior).

VideoEncoder doesn't signal that a frame has been dropped, there is simply no output. Authors can match timestamps if it wants to, but that's not very explicit. Happy to discuss something at a spec level.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants