add script for installing pandoc when not using conda

[andrewfulton9]

I've been trying to convert a project's docs to use an Jupyter notebook instead of a .rst file for its tutorial. The project uses tox and venv to build/test its docs and I had quite a bit of trouble getting pandoc installed into the venv on Linux Ubuntu 18.04. Pypandoc has a script to do this, but it doesn't look like there is much activity as far as releases for pypandoc (no accepted PRs since 06/2018 and no pip release since 04/2017). Their script in their master branch works, but their latest pip release is outdated for installing pandoc on linux machines. I am wondering if it would make sense to include a script like that in nbsphinx for people who can't use conda while Pandoc is still a nbsphinx dependency. I would have saved me quite a bit of time and looks like it could help with issues like #239

Thanks!

[mgeier]

I had quite a bit of trouble getting pandoc installed into the venv on Linux Ubuntu 18.04.

I understand that installing Pandoc might be complicated on some systems, but on most Linux distros it should be simple, shouldn't it?

For Ubuntu 18.04 there should be a package: https://packages.ubuntu.com/bionic/pandoc

I am wondering if it would make sense to include a script like that in nbsphinx

I'm not willing to maintain such a script. It sounds like the pypandoc people should maintain this script.
If pypandoc is abandoned, you can make your own fork and update the script.
Or you can talk to the current maintainers about taking the project over.

while Pandoc is still a nbsphinx dependency

That's an important point, I hope I can get rid of this dependency at some point. See #36.

[andrewfulton9]

I understand that installing Pandoc might be complicated on some systems, but on most Linux distros it should be simple, shouldn't it?

For Ubuntu 18.04 there should be a package: https://packages.ubuntu.com/bionic/pandoc

Sure, its simple to install on my own system, but that only helps me, it doesn't help anyone else who later wants to build the package's docs. In my opinion, if its a necessary requirement for the build, it should be easy to install in a python environment. With conda this isn't an issue, but with venv its not so straight forward, which is why I bring it up.

I also think, even if pypandoc was more actively maintained, including it in the environment for access to a single install script that's independent to the rest of pypandoc seems like a bloated and convoluted way get nbsphinx working via a venv environment.

Anyway, I understand were you are coming from, and can see why you wouldn't want the extra overhead for a temporary requirement. Thanks for taking the time to respond.

[mgeier]

Sure, its simple to install on my own system, but that only helps me, it doesn't help anyone else who later wants to build the package's docs.

According to your issue description it is about you and Ubuntu 18.04.

So I'm confused: is it "simple" or "quite a bit of trouble"?

In my opinion, if its a necessary requirement for the build, it should be easy to install in a python environment.

I agree, this would be great.
If the maintenance burden is reasonable.

With conda this isn't an issue, but with venv its not so straight forward, which is why I bring it up.

On Ubuntu?

I also think, even if pypandoc was more actively maintained, including it in the environment for access to a single install script that's independent to the rest of pypandoc seems like a bloated and convoluted way get nbsphinx working via a venv environment.

I'm not sure what you are suggesting, but I wouldn't automatically include it in the installation process anyway. If anything, I would provide it as an optional tool for those users who want to use it.

It doesn't make sense to run such a complicated script when you know Pandoc is already installed on your system, does it?

If there are users who don't know how to install Pandoc, they might as well install pypandoc and run the installation script.

Anyway, I understand were you are coming from, and can see why you wouldn't want the extra overhead for a temporary requirement.

It's not really about being temporary, the Pandoc dependency may stay there for a long time, who knows.

It's about the maintenance of such a highly complex script. If it were a few lines, no problem, but it has 180 lines.

It doesn't make sense to copy such a script between projects.
There should be one place where it is maintained and everyone should use the same maintained version instead of their own copy.

But if there is something I can do to make it easier to use such a script for nbsphinx, I'm very much open for it.

Or probably we should add pypandoc to the documentation?
But if it is unmaintained that's probably not so great?

[12rambau]

Old issue but perfectly relevant for my use case.
We want to drop myst-nb in favor of nbsphinx in the documentation of pydata-sphinx-theme mainly to remain compatible with pandas doc and Sphinx (see pydata/pydata-sphinx-theme#1474).

I wrote down a PR to rely on nbsphinx but we are facing the same problem as @andrewfulton9, the doc is now impossible to build for contributors that don't install manually pandoc, which is not acceptable if we want to be welcoming to new people.
pydata/pydata-sphinx-theme#1482

Since the opening of this issue, it seems pypandoc is back on track, last commit was in september and there are taking care of issues. They have the pypandoc_binary lib that includes the binary for standard platform but it is not compatible with nbsphinx.

which complains after installation with a:

Notebook error:
PandocMissing in examples/pydata.ipynb:
Pandoc wasn't found.
Please check that pandoc is installed:
https://pandoc.org/installing.html

Would you reconsider depending on pypandoc instead of vanilla pandoc ?

[mgeier]

They have the pypandoc_binary lib that includes the binary for standard platform but it is not compatible with nbsphinx.

The error comes from nbconvert (https://github.com/jupyter/nbconvert/blob/4c452c72d09a5f0b28d7fe7deb8b3f1be97a4a7b/nbconvert/utils/pandoc.py#L75-L78), so ideally this should be fixed there.

nbsphinx then calls pandoc in the same way as nbconvert does, so it can then be fixed in the same way.

Would you reconsider depending on pypandoc instead of vanilla pandoc ?

If I understand the above discussion correctly, this was never suggested.

But I would definitely consider depending on pypandoc if that helps users significantly more than it hurts.

Maybe this could also become an (optional) dependency of nbconvert, since this must have the same problem, right?

Either way: would that also work for arm64 macOS?

[12rambau]

Either way: would that also work for arm64 macOS?

I have 0 clue. I'm discovering pandoc all together at the moment

The error comes from nbconvert

Then that's why on most of my machines I don't have any issues as I'm always installing Jupyter. Let me check how they deal with it from their side.

[fzimmermann89]

We would like to use nbsphinx but still only require a pip install to also be able to build our docs.

One possible bandaid till a solution for #36 is found, would be to check in the 'pandoc' function if pypandoc can be imported.
If it can, use pypandocs function to find the pandoc binary, if not, just continue to call 'pandoc' and require pandoc to be on PATH (current behavior)

This way we could opt-in by installing pypandoc-binary and conda users can still use pandoc via comda without the need for pypandoc.

This would be a minimal change which I could commit to submit if @mgeier agrees and doesn't have the time to implement, but to review it.

Tackling #36 is unfortunately not feasible for me currently.

[mgeier]

This would be a minimal change which I could commit to submit if @mgeier agrees and doesn't have the time to implement, but to review it.

As I wrote above (#428 (comment)):

I would definitely consider depending on pypandoc if that helps users significantly more than it hurts.

I also wrote:

Maybe this could also become an (optional) dependency of nbconvert, since this must have the same problem, right?

Ideally, this should be handled in nbconvert, then it will (more or less) automatically also work for nbsphinx.

So I think this should be first suggested to nbconvert but if they are not willing to accept a PR, I'm willing to accept it here.

I'm interested in providing a smoother experience for users, so I'd be happy to review a PR!