Redundant rendering

[shashi]

I want to report an annoyance with the way many mime types are handled currently. In the IJulia kernel we have https://github.com/JuliaLang/IJulia.jl/blob/master/src/execute_request.jl#L24-L44 - which basically renders all possible mimetypes of an object and then sends all of them to the client and the client gets to pick the best one to show. I understand that this is because there are many consumers of the Jupyter messaging protocol and they want to be allowed to pick which mime type to show but this effectively makes Gadfly plots (which support rendering to HTML, SVG and PNG/JPEG) 3 times slower and 3 times as memory intensive, although only the HTML version is finally displayed in, say, the notebook interface. The problem is worsened in Julia since the first plot actually also compiles the code required for the rendering of other mime types which is a huge overkill and slows down the first plot considerably...

A good way to handle this is probably have the client send an ordered list of preferred mime types (probably as a separate message during start up) and the kernel can pick the best one that it can render to.

@jasongrout pointed out that, for archival notebooks, for example, you probably want all the mimetypes. Well yes, but I would love to have a way to speed it up if I am just using the notebook interface...

[Carreau]

We will be working on that by actually having the model on the server side, which will send only the require part to the browser.

As for which mimetype to actually compute, right now, usually libraries have global flag to enable/disable.

Any help to move things forward is of course appreciated.

[takluyver]

This is something we've thought about in other kernels as well. Until quite recently, the R kernel was drawing SVG, PNG, JPEG and PDF by default - the last especially made big chunks of binary data. We've now dropped JPEG and PDF from the defaults.

However, I don't want to go too far in the direction of only drawing the format the current client wants. I think it's a bad user experience if you have to explicitly think about whether this is an archival notebook or a quick-and-dirty exploration before making a plot. In a sense, the whole point of the notebook is to make a smoother transition from exploration to a more presentable document. So I think we need defaults that work reasonably well for both.

[JamiesHQ]

@takluyver @Carreau : has there been work on this since June 2015? What are the next steps to move this forward?

[gnestor]

It sounds to me like it's the library (e.g. pandas) that decides how big of a mime bundle to return and that users should be configuring that there (e.g. pandas.display.notebook_repr_html or pandas.display.html.table_schema). If the model were on the server/kernel as @Carreau mentioned, the notebook could dynamically fetch parts of the mime bundle depending on the context (like rendering an output or saving a notebook file), but currently it's all in the browser so users will be expected to manage that to a certain extent and library authors will be expected to manage that as well.

Another thing to consider is why libraries/kernels return multiple mime types for the same data in the first place. I think the primary reasons are for rendering a) in notebook environments that don't have the necessary extensions installed and b) using nbconvert. I think that binder is going to change some of this. binder makes it really easy to distribute reproducible notebooks. I think that people will people will start to share notebooks via binder because it effectively eliminates reason A above and it provides users with a live notebook vs. a static HTML notebook (like on GitHub). I don't have any answer for nbconvert...it could potentially render outputs using headless Chrome and capture static HTML for exporting to HTML, save screenshots for exporting to PDF, etc. @mpacer Is there any plan to support custom mime types in nbconvert?

[gnestor]

Somewhat related: @Carreau created a prototype that allows users to toggle between different available mime types for an output 👍: #2769

[Carreau]

If the model were on the server/kernel as @Carreau mentioned, the notebook could dynamically fetch parts of the mime bundle depending on the context (like rendering an output or saving a notebook file)

Yes, but that won't reduce the computation time.

I think that binder is going to change some of this. binder makes it really easy to distribute reproducible notebooks

That might be the case for some small examples, but notebook storing the data is critical in other case, if you need credential to access data, or if you run a simulation on HPC, you can't possibly ask binder to rerun that. On of the classic example would be a notebook that analyses the last 100 Tweets, about #hastag the result will depends on when you run the notebook, so your mimetypes have to be computed at execution time.

We are still thinking about how to make that easier/better.