Sphinx should be aware of all.py to find its links

[hivert]

Though sphinx is perfectly working with target in the local module he isn't
able to find reference target from other modules even if they are exported in
all.py. For example, if I want to link Parent from anywhere but parent.pyx, I
have to write the full path (ie. :class:`~sage.structure.parent.Parent`)
even if it is imported in my module. I find this extremely annoying since, in
the task of improving the category doc, I'm setting up a lot of huge cross
references such as:

    :meth:`Algebras.ParentMethods.algebra_generators()
    <sage.categories.algebras.Algebras.ParentMethods.algebra_generators>`.

I would be very happy if I had to write only

    :meth:`Algebras.ParentMethods.algebra_generators`

The following patch should solve this issue

I set up intersphinx so that links to Python's doc are correctly
resolved. The patch
attachment: trac_9128-intersphinx_python_database-fh.patch contains Python's
crossref database downloaded from http://docs.python.org/objects.inv I'm also
using intersphinx to solve links to the reference manual from the other
documents.

New option for docbuild

I added the option --warn-links to the documentation build command as in

    sage -docbuild --warn-links reference htm

When the option is used, Sphinx will issue a warning, whenever a link is not resolved.

Extra features

Moreover, I add a reference target to every automatically created .rst
file associated to python source files. It can by used to set-up a link toward
the file and thus get the title rather than the module. For example
:ref:sage.categories.primer setup a link to the help page with title
"Elements, parents, and categories in Sage: a (draft of) primer" rather
than
sage.categories.primer which you get with the link
:mod:sage.categories.primer.

Apply both:

Note: order is irrelevant

• attachment: trac_9128-intersphinx_python_database-fh.patch
• attachment: trac_9128-sphinx_links_all-fh.patch
• attachment: trac_9128-MANIFEST-fh.patch

Depends on #11251
Depends on #12490
Depends on #12572

CC: @nthiery @nexttime @novoselt @mwhansen

Component: documentation

Keywords: Sphinx links

Author: Florent Hivert

Reviewer: Andrey Novoseltsev, Nicolas M. Thiéry

Merged: sage-5.0.beta8

Issue created by migration from https://trac.sagemath.org/ticket/9128

[hivert]

comment:1

The patch here is a prototype, not yet ready for review. Comments or suggestions are mostly welcome.

[hivert]

Description changed:

--- 
+++ 
@@ -17,3 +17,11 @@
 ```
 The following patch should solve this issue
 
+I also added an option `SAGE_DOC_WARN_DANGLING_LINKS` which raises some
+warnings if some links aren't resolved. For example:
+
+```
+WARNING: symbol :meth:`ModulesWithBasis.HomCategory.ElementMethods.on_basis` linked from sage.categories.modules_with_basis is defined in sage.categories.modules_with_basis but not documented
+WARNING: undefined symbol :class:`CartesianProducts.ParentMethods` in module sage.categories.modules_with_basis
+```
+Note those two warnings aren't raised because of a wrong doc but due to #9107.

[hivert]

comment:4

The submitted patch seems to behave as I want. So I put needs review. However if I receive some good advice on sage-devel or sphinx-users I may still change it. Anyway, Please comment.

[novoselt]

comment:6

I have just upgraded an installation of sage-4.4.2 to 4.4.3, applied this patch, set SAGE_DOC_WARN_DANGLING_LINKS to 1, and then got the following error:

novoselt@sage:/scratch/novoselt/sage/devel/sage-main$ ../../sage -docbuild reference html
sphinx-build -b html -d /mnt/usb1/scratch/novoselt/sage/devel/sage/doc/output/doctrees/en/reference    /mnt/usb1/scratch/novoselt/sage/devel/sage/doc/en/reference /mnt/usb1/scratch/novoselt/sage/devel/sage/doc/output/html/en/reference
Running Sphinx v0.6.3
loading pickled environment... done
building [html]: targets for 798 source files that are out of date
updating environment: [config changed] 802 added, 0 changed, 0 removed
reading sources... [100%] structure
/mnt/usb1/scratch/novoselt/sage/local/lib/python2.6/site-packages/sage/schemes/elliptic_curves/sha_tate.py:docstring of sage.schemes.elliptic_curves.sha_tate.Sha.bound_kato:12: (WARNING/2) Definition list ends without a blank line; unexpected unindent.
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [  0%] coercion
Exception occurred:
  File "/mnt/usb1/scratch/novoselt/sage/devel/sage/doc/common/conf.py", line 444, in find_sage_dangling_links
    fromdocname = getattr(sys.modules[modname], basename).__module__
KeyError: None
The full traceback has been saved in /tmp/sphinx-err-Unu279.log, if you want to report the issue to the author.
Please also report this if it was a user error, so that a better error message can be provided next time.
Send reports to sphinx-dev@googlegroups.com. Thanks!
Build finished.  The built documents can be found in /mnt/usb1/scratch/novoselt/sage/devel/sage/doc/output/html/en/reference
novoselt@sage:/scratch/novoselt/sage/devel/sage-main$

Then I removed this patch, built documentation successfully, pushed this and some other patches and now it seems to work fine. Perhaps it was an unreproducible glitch unrelated to the patch. In general it seems to work as expected and showed me a couple of mistakes in my modules.

I hesitate to switch status to positive review because of the error above and because I don't really understand the code, but I think that this is a great addition and will use this patch from now on!

[novoselt]

comment:7

I tried the same thing on my own computer - upgraded from 4.4.2 to 4.4.3, applied this patch and tried to build documentation (without setting the environment variable this time). Same error repeatedly, but after popping the patch out everything goes OK. So something has to be done ;-)