Add hardware acceleration to video decoding

[rvillalba-novetta]

• Add optional dictionary parameter to input open allowing the user to pass setting for hardware acceleration
• Add nvidia libraries to the ffmpeg build if available

[mikeboers]

Note that this is related to #307.

[mikeboers]

I've rebased this onto the current develop. It is in the hwaccel branch.

[hunterjm]

Any ideas on when this will be updated with the current branch and merged?

[jlaine]

@mikeboers I'm totally confused about this PR. If the most up-to-date code is on your branch, let's close this PR and open a new one based on your branch?

[jlaine]

I have rebased this PR on top of develop, pushed it to @rvillalba-novetta 's repo and killed the hwaccel branch on the main repo. The status seems to be : it only builds with ffmpeg 4.x

[mikeboers]

Thanks for that @jlaine. Without putting much effort in (because as you can tell I'm not at the moment), can you identify why it is FFmpeg 4.x only?

[jlaine]

It doesn't even compile on older ffmpegs so I'm wondering if it's an entirely new API

[mikeboers]

Looks like the headers go back to at least 3.3.

[mikeboers]

avcodec_get_hw_config does not exist < 4.

[mikeboers]

As far as the example is concerned, it replaces this (annoying) function -> https://www.ffmpeg.org/doxygen/3.4/hw__decode_8c_source.html#l00047

[mabl]

I have had great results with this patch. I additionally added hwaccel to CodecContext.create and Codec.create, to allow creation of hardware accelerated Codecs without a container.

[mikeboers]

Now that we've dropped FFmpeg < 4.0, this is more approachable as is. The tests pass. So now (at some point) we can decide if we like the API and how it is implemented.

[mikeboers]

Actually... I think I'm done with this at this point because:

1. the ffmpeg CLI isn't even able to use hardware decoding on my workstation, so this is a pain to work on, and
2. man ffmpeg says:

Note that most acceleration methods are intended for playback and will not be faster than software decoding on modern CPUs. Additionally, ffmpeg will usually need to copy the decoded frames from the GPU memory into the system memory, resulting in further performance loss. This option is thus mainly useful for testing.

Since it is unreasonable to think that anyone is decoding with PyAV just for playback, this seems pretty pointless to keep fighting.

Thanks, everyone, for your time on this, but I'm getting off here.

[wegel]

Hmm, hardware acceleration in the context of PyAV for me is mainly for encoding, not decoding. For encoding there's a substantial performance gain, and you can do multiple encodes in parallel if you have the correct hardware.

[razvanphp]

Not to open a new "question-like" issue, is it expected we can use hardware accelerated encoders? For example, h264_v4l2m2m or h264_omx?

With PyAV 8.0.1 the first one triggers:

...
    packages = self.codec.encode(frame)
  File "av/codec/context.pyx", line 458, in av.codec.context.CodecContext.encode
  File "av/codec/context.pyx", line 464, in av.codec.context.CodecContext.encode
  File "av/codec/context.pyx", line 275, in av.codec.context.CodecContext.open
  File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument; last error log: [h264_v4l2m2m] can't configure encoder

and it's used like:

codec = av.CodecContext.create("h264_v4l2m2m", "w")
codec.width = frame.width
codec.height = frame.height
codec.pix_fmt = "yuv420p"
codec.time_base = fractions.Fraction(1, MAX_FRAME_RATE)
codec.options = {
    "h264_profile": "baseline",
    "level": "31",
    "tune": "zerolatency",
}

packages = self.codec.encode(frame)

I also tried h264_omx on raspberry Pi4, and while it works in ffmpeg directly, on pyav it hangs the app unresponsive.

Any suggestions? There is also nothing in the documentation about this flow...

Thank you!

[pallas]

FWIW, I do want to receive a h264 stream from a remote machine and directly render on a Raspberry Pi. I may take a shot at this if PyAV sees it as out of scope.

[vade]

Theres other use cases where hardware decode is very useful outside of playback - decode to hardware accelerated memory.

For my use case I would like to be able to decode into memory backed by the GPU to send as a tensor for inference for machine learning. PyAV is perfect as a shim to libAV to have finer grained frame access and in process control - however without GPU accelerated memory its not quite as appealing.

Note this is also very helpful for server side on demand rendering on headless GPU instances - decoding directly to a texture is quite nice. We did used C++ for this project using LibAV with nvdec on AWS hardware : https://rarevolume.com/work/reuterstv/ - it would have been much slower (and more expensive) sans libAVs nvdec implementation.

Being able to use Python + libav + nvdec / nvenc would allow a a lot of nice optimal code paths outside of rendering:

• encoding (as mentioned)
• rendering
• machine learning inference
• machine learning training
• automated quality assessment
• hardware accelerated transcoding
• GPU accelerated computer vision
  (and so on)

Thank you.

[vade]

Actually... I think I'm done with this at this point because:

1. the ffmpeg CLI isn't even able to use hardware decoding on my workstation, so this is a pain to work on, and
2. man ffmpeg says:

Note that most acceleration methods are intended for playback and will not be faster than software decoding on modern CPUs. Additionally, ffmpeg will usually need to copy the decoded frames from the GPU memory into the system memory, resulting in further performance loss. This option is thus mainly useful for testing.

Since it is unreasonable to think that anyone is decoding with PyAV just for playback, this seems pretty pointless to keep fighting.

Thanks, everyone, for your time on this, but I'm getting off here.

Hi - I was able to do some tests using VPF (Nvidia's) new video performance framework in Python and test these claims.

They dont hold up. Not even close.

PyAV on a Google Colab machine for a h.264 QuickTime has the following performance:

One comment for those in the public - while the shared Google Colab environment can have different load so performance benchmarking is difficult, for the same h.264 QuickTime mov file -

LibAV CPU:

Decode took: 0.9556884765625 seconds
fps 92.08021458679269
3.8401634946991954 x realtime

VPF try on a T4 on collab GPU instance try one:

Decode took: 0.42092251777648926 seconds
fps 209.06460520301312
8.71894433062566 x realtime

Try 2 (less contention perhaps?)

Decode took:  0.2164461612701416 seconds
fps 406.5676170166362
16.955717664216532 x realtime

This is rendering back to the CPU, so it includes GPU to CPU transfer. That is a non trivial performance increase with NVDec.

[vade]

my suggestion to the VPF authors at Nvidia is to leverage PyAV for Demuxing and have them build a special NVEnc NVDec packet decoder which can vend surfaces (on the gpu) or frames (CPU backed memory which has been read back)

This would mean HWAccel is ignored - so other existing back ends dont get support.
However, it does mean that Python could vend GPU backed frames using PyAV and VPF together - and keep PyAV clean of the complexities of HWAccel support (which is admittedly tricky)

I do feel however its a murky fix and not really inline with having LibAV fully ported to python. Given the above perf delta - which is real, demonstrable (look at new updates from Adobe switching to use NV encoder and decoder) as well as Apple using HW Accel in Video Toolbox via the T2 and AMD chipsets - the gains are tremendous for pro workflows.

I highly suggest this is reconsidered.

Can I help somehow?

[philipp-schmidt]

Any news regarding the hardware acceleration? I'm currently using VPF to do this but I'm having issues with rtsp transport of IP cameras.

Could someone outline how a solution with VPF and PyAV in tandem would look like?

Essentially I want PyAV to handle transport and container related stuff and then let VPF handle the H.264 decoding. Is this a valid use-case for PyAV?

NVIDIA/VideoProcessingFramework#132

[philipp-schmidt]

VPF authors have found a way to combine this and are waiting for bitstream support in PyAV

NVIDIA/VideoProcessingFramework#99 (comment)

[kennethtang4]

I think one of the main reasons to use hardware acceleration is to offload the processing power to GPU and keep the CPU free, not only about the processing time only. CPU workload dropped from 30+% to less than 2% after enabling hardware acceleration for decoding 1080P@30. I did that by using C++ to call FFmpeg libraries and I would say it rather simple after all as long as you know the concept behind it and which function to call. Though, I am using the precompiled FFmpeg libraries (libavcodec, libavformat etc…) as it is tedious to compile every dependencies for each different hardware.

I am more than happy to help if Mike is up to it. Could setup a remote workstation for you to test things out if that is you concern.