Feature request: loading jpeg from memory (ndarray and Tensor)

[vadimkantorov]

Probably jpeg_mem_src would help (accepts a offset and count).

[willprice]

This is addressed in #32/#35

[vadimkantorov]

Is there support for batch decoding (because of variable-sized inputs, either extra masks or lengths argument is going to be required, similar to padded sequences; or only accepting input as list; or nested/jagged tensor)

[willprice]

Can't you just create a view on the buffer to mask out the portion that corresponds to a single jpeg?

[vadimkantorov]

The motivation for batch decoding isn't just avoiding selecting out single jpeg's. It's for saving on parallel processing by making it more coordinated and having more control over it.

E.g. modern audio codecs support batched decoding for that reason: https://mobile.twitter.com/neilzegh/status/1426122384357134337

nvjpeg might support it as well

[vadimkantorov]

Also, if accimage is a thin wrapper to libjpeg(-turbo), then maybe torchvision also wraps it directly these days?

[willprice]

The motivation for batch decoding isn't just avoiding selecting out single jpeg's. It's for saving on parallel processing by making it more coordinated and having more control over it.

Sure, but accimage is a thin veneer around libjpeg-turbo and intel's IPP image processing routines to mimic the PIL Image API. It doesn't look as if IPP supports batched processing. It parallelises over the rows of the image IIRC. Likewise, I don't think the libjpeg API provides batch decoding.

Also, if accimage is a thin wrapper to libjpeg(-turbo), then maybe torchvision also wraps it directly these days?

That's besides the point, as accimage can be used outside of torchvision. If you're interested in what torchvision supports, then check out https://pytorch.org/vision/stable/io.html#image. They use the libjpeg API too, it's not clear to me looking at pytorch/vision#2291 whether they build against libjpeg-turbo by default or not. At the very least, it shouldn't be too difficult to build torchvision from source against libjpeg-turbo if you wish to use torchvision.io.read_image instead. They too have a decode from bytes function (https://pytorch.org/vision/stable/io.html#torchvision.io.decode_image) although this isn't batched.

nvjpeg might support it as well

Accimage will not support nvjpeg, building a PIL.Image compatible interface backed by GPU memory is out of the scope of this project.

E.g. modern audio codecs support batched decoding for that reason:

https://mobile.twitter.com/neilzegh/status/1426122384357134337
This is a neural network based audio compressor, it makes sense it'd be batched. Older APIs like libjpeg weren't designed with this in mind. They parallelise the decoding of the JPEG using SIMD instructions (the image operations in accimage are SIMD accelerated using intel IPP)