Ability to work along side PyAV?

[vade]

Describe the bug
Hello - VPF appears to be functioning decently in a simple test case - however it hides / abstracts / removes (?) a lot of libAV functionality in python behind a simple interface.

Is it possible to use pyAV to vend compressed packets of HEVC or AVC data and send to VPF ? This would allow for nice flexibility of having most of libAV at your disposal (audio, re-muxing, time stamps, etc) while having access to the speed of VPF .

Am I misunderstanding the API or is this possible today with VPF?

Thank you.

[rarzumanyan]

Hi @vade

Hello - VPF appears to be functioning decently in a simple test case - however it hides / abstracts / removes (?) a lot of libAV functionality in python behind a simple interface.

VPF uses ffmpeg (not libAV) under the hood to obtain the Annex.B elementary bitstream from the input URL (file, network, etc.).
Rest is done using Video Codec SDK and CUDA.

Technically it's easy to take compressed video frames from outside and feed them to HW.
This is the function which accepts compressed video frame and kicks off decoding:

VideoProcessingFramework/PyNvCodec/src/PyNvCodec.cpp

Lines 452 to 467 in ae46083

	static Surface *getDecodedSurface(NvdecDecodeFrame *decoder,
	DemuxFrame *demuxer,
	bool &hw_decoder_failure) {
	hw_decoder_failure = false;
	Surface *surface = nullptr;
	do {
	/* Get encoded frame from demuxer;
	* May be null, but that's ok - it will flush decoder;
	*/
	auto elementaryVideo = getElementaryVideo(demuxer);
	/* Kick off HW decoding;
	* We may not have decoded surface here as decoder is async;
	* Decoder may throw exception. In such situation we reset it;
	*/
	decoder->SetInput(elementaryVideo, 0U);

It doesn't really matter where the elementaryVideo comes from.
I'm ok to extend the VPF interface with this as long as desing stays consistent.

The easiest approach to this IMO is to add another PyNvCodec class ctor which won't create demuxer and add couple more decoding methods which accept numpy arrays with Annex.B compressed frames.

[vade]

hi @rarzumanyan - thanks for the swift response.

I totally get that, ive used NVEnc / NVDec in C++ with LibAV in the past - however on the python side of things the work you've done to make a CUDA / GPU backing available to publish to other tensor tooling looks really nice, and it would be super nice to:

• be able to use LibAV like interface to handle container parsing, metadata, time code,
• use LibAV's codecs for audio processing, sample packet handling of other unsupported codecs outside of VPF
• use VPF's NVEnc and NVDec front ends to manage GPU accelerated decode
• use VPF's direct to surface, and surface to tensor support
• have VPF's hardware accelerated color space / format conversions

The PyAV authors dont appear to be interested in supporting the full suite of HWAccels that LibAV integrates (a shame) - but I think having VPF python API handle something like:

• Current use case - pass in a supported container/stream with a supported codec and just get frames
• New : Manually create a decoder or encoder context and use that to parse manually demuxed packets from a container that PyAV/LibAV vends to a the decoder to output a surface or a frame.
• New : Same for encoding - pass in a surface or some such and produce a packet that can be consumed and muxed by PyAV

This would then support surface to tensor functionality and keeping things on the GPU.

This is helpful for pro video workflows - its doable in C++ but, we have Python, and 2020 is already bad enough haha :)

Does Nvidia take 3rd party PR's? Im happy to try to wire something up but im unfamiliar with cython / python dev. In short, let me know how I can help.

I have a Google Colab notebook which demonstrates setting up a VPF GPU accelerated decoder with custom compilation and benchmarking against LibAV / pyAV - im happy to donate to the project as an example / QuickStart.

[vade]

One comment for those in the public - while the shared Google Colab environment can have different load so performance benchmarking is difficult, for the same h.264 QuickTime mov file -

LibAV CPU:

Decode took: 0.9556884765625 seconds
fps 92.08021458679269
3.8401634946991954 x realtime

VPF try on a T4 on collab GPU instance try one:

Decode took: 0.42092251777648926 seconds
fps 209.06460520301312
8.71894433062566 x realtime

Try 2 (less contention perhaps?)

Decode took:  0.2164461612701416 seconds
fps 406.5676170166362
16.955717664216532 x realtime

[rarzumanyan]

@vade

Does Nvidia take 3rd party PR's?

Yes.
There are 4 contributors so far and only one of them works for Nvidia :)

New : Same for encoding - pass in a surface or some such and produce a packet that can be consumed and muxed by PyAV

This is already done. PyNvEncoder class outputs AnnexB encoded frame which you may mux using PyAV.
BTW you can see this in encoding samples whici save output to Annex.B format (you can play it with VLC).

The feature you request is relatively simple, I can prepare a patch and push it so separate branch.
It would be very nice if you can test it and brush up the code.
VPF is more like a hobby project for me which I'm allowed to publish so often I don'e have enough time because of my main job duties.

[rarzumanyan]

@vade

I've pushed a commit to issue_99 branch.
The code compiles but I can't test it as I don't know how to demux from Python.
Please use this commit as a draft, feel free to ask if you have any questions / issues.

[vade]

Thank you so much, this is exciting. I sincerely appreciate the prompt reply. Ill write up a little demux harness via PyAv and compile off of your branch. Seriously appreciate it!

[rarzumanyan]

@vade

You're welcome.
I also don't like the idea of keeping the decoder & demuxer together but that's what people prefer as the single most popular use case is to decode from RTSP camera and apply ML algos to the video frames.
So I hope pyAV demuxer could save me some pain because issues often happens outside the video decoding but somewhere in the RTSP lands.

[vade]

Totally! Im in the same boat. Im close to having something working - check out this collab notebook

https://colab.research.google.com/drive/1LMPp2zqCjuUdmoM4XUwTHzmsEGZxl3W_?usp=sharing

Im able to vend packets from PyAV to an instantiated NVDec instance in python via the new issue_99 codebase using nvc.cudaVideoCodec_H264 initializer, but im having trouble passing in data in the right format from a libav packet into an np array.

      3 def decode_vpf(packet):
----> 4   np_encoded = np.array(packet.to_bytes(), dtype=np.uint8)
      5   np_decoded = np.ndarray(shape=(3,2), dtype=np.uint8)
      6 

ValueError: invalid literal for int() with base 10: b'\x00\x00\x00\x14\x06\x05\x10\xb9\xed\xb90]!Kq\x83q,\x10\xa3\x14\xbb)\x80\x00\x02\xc6\xd9%\xb8\x00@\x00\x06\xff\xe3\x86}:\xb3\xfb\x84\xa3\x1fcc!\xa2\xbc\x06Z-\x7f\xb3]%\xd5\xdei\xa8j\x930d`ND\xb3\xa6```

Really appreciate your prompt replies!

[vade]

A touch closer - one note, in order to resolve the above issues, im doing something along the lines of:

nvDec = nvc.PyNvDecoder(nvc.cudaVideoCodec_H264, 0)

#todo: make a pool / list of n writable np_array destination frames I can recycle

def decode_vpf_frame(packet):
  # not sure how I should be formatting my input to DecodeSingleFrameFromPacket ? 
  np_encoded = np.frombuffer(packet.to_bytes(),  dtype=np.uint8)
  # np_encoded.setflags(write=1)

  np_decoded = np.zeros(1920*1080*3, dtype=np.uint8)
  np_decoded.setflags(write=1)

  nvDec.DecodeSingleFrameFromPacket(np_encoded, np_decoded)
  return np_decoded

however it appears both nvDec.DecodeSingleFrameFromPacket and nvDec.DecodeSingleSurfaceFromPacket requires the init np.array to be writable.

I cant appear to set the writable flag directly on the np_encoded buffer, but I can if I copy it. However doing that seems to stall the system?

[rarzumanyan]

@vade
DecodeSingleFrame and DecodeSingleFrameFromPacket take numpy arrays by reference and write raw frames in-place.

The reason may seem weird, but if I return numpy array from these methods with move constructor, it will make multi-threaded python scripts crush.

I’ve wasted almost a week debugging this some time ago. The root cause is unknown for me, I’ve just found that any py::array_t constructor call from within C++ VPF internals cause multi threaded pythons scripts to crash.

However, the numpy arrays with encoded frames don't need to be modified from within VPF - please add const qualifiers to the patch if you want them to be explicitly constant.

[rarzumanyan]

@vade
I've added const qualifiers on C++ side - shall be fine from Python now.
Please check the latest commit in issue_99 branch.

[vade]

Ah! I want sure how python arrived at the writability flags. const makes sense in retrospect. Thank you again for such availability - I really appreciate it.

your fix above resolves the writability warning/error I receive, however, im now apparently crashing google Colab's runtime kernel ha.

Ive reset the runtime to ensure the right .so is loaded, etc. Ill try a factory reset and see. This is close!