[RFC] Simplified 'sphinx-build'

[stephenfin]

Feature or Bugfix

• Feature

Purpose

Make sphinx-build easier to call by moving configuration of the build directory to the configuration file and auto-detecting the source directory. This will allow most users to call sphinx-build without any arguments.

Detail

• sphinx-build: Configure outdir from conf.py

  This is part one of two of the "make 'sphinx-build' easier to call" effort. In this part, we allow users to define the build directory in the 'conf.py' file. This means a user no longer needs to pass the option via the command line, though they can continue to do so if they wish.

  We use the behavior of the setuptools extension, whereby the results are output to a subfolder in the outdir named after the code of the builder. For example, if using the 'html' builder with 'output_dir = "test"', files would be written to 'test/html'.

• sphinx-build: Guess sourcedir if none provided

  This is part two of two of the "make 'sphinx-build' easier to call" effort. In this part, we attempt to guess where the source directory is if the user does not provide one. This is possible because many users use one of three directory names - doc, docs, or Documentation - for their documentation. We check these, along with a possible 'source' subdirectory, which is based on Sphinx's own defaults, for a 'conf.py' file. Is one is found, we assume this is the documentation source. This means a user no longer needs to pass any argument to 'sphinx-build' is they do not wish to.

  This is, of course, best-effort, and users can continue to provide these options manually if they so wish.

You can validate this right now by pulling down the pull request and running sphinx-build without any arguments. The required change has been made to conf.py.

[stephenfin]

This was previously proposed at #4320, but there were some discussion around it and not everyone agreed. Having continued to use Sphinx extensively, I'd like to repropose this in the hopes that I can convince people that it's a good idea. With this change, the typical Sphinx invocation goes from something like this:

sphinx-build -b html doc/source doc/build

to

sphinx-build

Which is a heck of a lot easier to remember, if you ask me. I realize there's a bit of magic here, but I think it's a good thing for users. The only other option I can see is to modify #6938 so that the new sphinx build command does not take positional arguments and instead you must configure the output and source directory using flags (e.g. -i sourcedir -o outputdir). I'd even be happy having to pass the sourcedir, but I really do think outputdir (and later, default builder(s)) is something we should be able to store in conf.py.

[tk0miya]

+1 for "Make sphinx-build easier to call". In other words, +1 for making command-line arguments optional. But I've not still understand the advantage of moving output directory to conf.py. I think the output directory is not portable for each developers...
In addition, I'm also worry about aboslute path might harm somebody (ex, windows platform, security attack (like ../../../../etc) and so on.
IMO, default output directory is better. For example, _build/{builder_name} is good to me.

Next, about guessing srcdir, -1 to this. Only I can accept is using current directory as srcdir. It's not magical to me.

[stephenfin]

+1 for "Make sphinx-build easier to call". In other words, +1 for making command-line arguments optional. But I've not still understand the advantage of moving output directory to conf.py. I think the output directory is not portable for each developers...

To be clear, I'm not suggesting we remove the ability to set the output directory using a configuration option. Instead, I'm suggesting we simply provide an additional way to configure this. Personally, I don't think the output directory is any different from other configuration, such as project name or HTML theme. If you are building documentation, it is likely that you would always want to output that to the same place. It would be silly to have to configure project name from from the command line every time you called sphinx-build (or, soon, sphinx build) and it seems silly to have to configure the output directory this way.

In addition, I'm also worry about aboslute path might harm somebody (ex, windows platform, security attack (like ../../../../etc) and so on.

This might be true, but isn't this the case already if you build documentation using a tox target (tox -e docs), a Makefile target (make html) or using the distutils entry point (python setup.py build_sphinx)? In each of these cases, we're already storing the output directory as static configuration but in a different place: as a command line flag, as a Makefile variable, and as a setting in setup.py, respectively. Why not simply store it in the same place as all the other Sphinx configuration?

IMO, default output directory is better. For example, _build/{builder_name} is good to me.

I don't think that offers enough flexibility, given sphinx-quickstart already offers the way to optionally split the doctree between separate source and output directories. Again, it should be totally possible to override the default configuration but I'd like to make that default configurable.

Next, about guessing srcdir, -1 to this. Only I can accept is using current directory as srcdir. It's not magical to me.

I still like this idea, but it seems I'm the only one that does 😄 My ultimate goal is to be able to call sphinx-build (or sphinx build) and everything will just work, and I don't consider checking well known directories to be too magical. However, making the output directory is more important to me and this becomes less important if we're adding a new sphinx build command since I can convert the outputdir argument for that to a flag. I can forget about this.

So, here's my proposal.

• Add output_dir config option for conf.py
• Make the outdir argument for sphinx-build optional
• When adding the sphinx build command, add a -o outdir option instead of the outdir positional argument, so the command would be sphinx build <source_dir> [<file> ...].

Given the above, does this make sense?

[jakobandersen]

I agree that the command line interface should not be more complicated than necessary, I am not sure if this is a good simplification.
Source dir: Guessing the location is (usually) harmless. If not found, nothing happens. Though I prefer that the guess work is not too complicated, and I lean towards using the current directory as the only guess.
Output dir:

• I dislike doing "guessing" here. Sphinx generates more than a single file, so if the user is not aware of exactly which files are generated the program may in the worst case degenerate into a tar bomb.
• Having the output dir in the configuration makes the potential "tar bomb problem" worse. The user is responsible for cleaning up so the amount of surprises should be minimal.
• Defaulting to a path outside the current tree (e.g., ../build) is unacceptable to me.
• It generally feels very wrong to specify the output dir in the configuration file. It is a fundamentally different property: the package author writes the configuration, the user (perhaps through a build system) specifies where the output goes.

If there should be simplification of the source/output paths I suggest the following:

• Source dir defaults to the current dir.
• If the source dir was not specified by the user, then output dir is allowed to default to something like _build. Otherwise the user must also specify the output dir.

The source dir is somewhat in control of the package author so they know not to use _build (or whatever dir) for something else.

[stephenfin]

I agree that the command line interface should not be more complicated than necessary, I am not sure if this is a good simplification.
Source dir: Guessing the location is (usually) harmless. If not found, nothing happens. Though I prefer that the guess work is not too complicated, and I lean towards using the current directory as the only guess.

As above, this is the way I'm leaning now too since I think I'm alone in not being bothered by the auto-guessing behavior.

Output dir:

• I dislike doing "guessing" here. Sphinx generates more than a single file, so if the user is not aware of exactly which files are generated the program may in the worst case degenerate into a tar bomb.

What guessing do you refer to? The choice of output directory is clearly defined: either the sphinx-build argument is used, if provided, else the output directory option from conf.py is used. If neither is provided, the program exits. Both are obvious to the user, since they explicitly passed the former [*] and the latter is a clearly defined config option in the sphinx configuration file.

[*] Well, as noted above, most projects I interact with set these by default and hide them inside a tox.ini file or a Makefile or something else.

• Having the output dir in the configuration makes the potential "tar bomb problem" worse. The user is responsible for cleaning up so the amount of surprises should be minimal.

Again, this isn't doesn't seem to be a problem because if it affected the user, it would also affect the package maintainer who set this default. If the user would really nervous, just keep passing an output directory as before. I am not removing the ability to do this.

• Defaulting to a path outside the current tree (e.g., ../build) is unacceptable to me.

Why? Any user is free to override this but the project maintainer(s) get to provide a default that works best for them.

• It generally feels very wrong to specify the output dir in the configuration file. It is a fundamentally different property: the package author writes the configuration, the user (perhaps through a build system) specifies where the output goes.

Again, why? I'm targeting the 90% use case here. In almost every project I work on, a docs tox target is defined to build documentation and almost every user (including the various CI jobs) use this default configuration (which is implicitly encoded in other places such as .gitignore too, fwiw). This is good behavior. Most users just want to build their docs and they don't care where they go so long as that output directory is discoverable. For users that do want to override this output directory, they can continue to do so, just as they can choose to override almost anything else about the configuration. Perhaps I'm running into a barrier with how people are used to other projects working, but allowing us to provide a configurable default makes sense to me, as does storing this configuration in the same place as all other configuration.

If there should be simplification of the source/output paths I suggest the following:

• Source dir defaults to the current dir.

Seems fair.

• If the source dir was not specified by the user, then output dir is allowed to default to something like _build. Otherwise the user must also specify the output dir.

As noted previously, that breaks users that use the "separate source and build directories" option that sphinx-quickstart provides and is a non-starter for that reason.

The source dir is somewhat in control of the package author so they know not to use _build (or whatever dir) for something else.

All in all, this is entirely optional behavior that is wanted by some people and can be safely ignored by others, so I'm not sure why we wouldn't want to do this, heh.

[tk0miya]

@stephenfin Thank you for your clear explanation. I'm perfectly understanding your idea. I'm happy to agree with you about srcdir not guessing. But I'm still worrying about output_dir in conf.py. I must admit that it is almost same to place it on conf.py or build script (ex. Makefile, build.sh, etc.). But I've felt bad smells from its design. Sorry for I can't describe it to you well. Please give time to consider why I feel so bad to your idea.

BTW, we have to clear a term in this topic to discuss more deeply. It's "output_dir". At present, we have two types of output_dir. First is a directory used to generate output files just inside it. _build/html and _build/latex are one of them. And second is a directory used to generate former directories. _build is used in former example. In other words, output_dir in "non-make-mode" is former one, and output_dir in "make-mode" is latter one.

I guess output_dir in conf.py is latter one. So sphinx build -b html will generate files on [output_dir]/html. Is this right? On the other hand, a directory specified by -o option is former one. So sphinx build -b html -o [output_dir] will generate files just on [output_dir]. Is my understanding right?

[stephenfin]

BTW, we have to clear a term in this topic to discuss more deeply. It's "output_dir". At present, we have two types of output_dir. First is a directory used to generate output files just inside it. _build/html and _build/latex are one of them. And second is a directory used to generate former directories. _build is used in former example. In other words, output_dir in "non-make-mode" is former one, and output_dir in "make-mode" is latter one.

I guess output_dir in conf.py is latter one. So sphinx build -b html will generate files on [output_dir]/html. Is this right? On the other hand, a directory specified by -o option is former one. So sphinx build -b html -o [output_dir] will generate files just on [output_dir]. Is my understanding right?

Correct. This is similar to the behavior of make mode, as you note, and seems sane when you consider the default support for multiple builders. Again, if someone wants to output to a specific location, they can choose to pass the config option.

[woutervh]

This merge request is open for 4 years now. What is blocking this?

[AA-Turner]

What is blocking this?

That it wasn't accepted as proposed. I intend to add support for configuring source and output directories at some later date, but I agree with the concerns about 'guessing' (this could e.g. be done in quickstart).

However, thank you for the prompt, it makes sense to close this PR which is now out-of-date.

A