sagelib setup.py: Dependencies on header files of packages

[mkoeppe]

Rebuilds should be triggered when headers of relevant packages change.

In setup.py (after #29702, sage_setup.command.sage_build_cython) we have:

# Do not put all, but only the most common libraries and their headers
# (that are likely to change on an upgrade) here:
# [At least at the moment. Make sure the headers aren't copied with "-p",
# or explicitly touch them in the respective spkg's spkg-install.]
lib_headers = { "gmp":     [ os.path.join(SAGE_INC, 'gmp.h') ],   # cf. #8664, #9896
                "gmpxx":   [ os.path.join(SAGE_INC, 'gmpxx.h') ],
                "ntl":     [ os.path.join(SAGE_INC, 'NTL', 'config.h') ]
              }

Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ).

(from #29411)

CC: @dimpase @EmmanuelCharpentier @orlitzky @kliem @slel

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/29711

[mkoeppe]

Description changed:

--- 
+++ 
@@ -12,6 +12,6 @@
                 "ntl":     [ os.path.join(SAGE_INC, 'NTL', 'config.h') ]
               }
 ```
-Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ)..
+Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ).
 
-
+(from #29411)

[orlitzky]

comment:2

What you really want to do here is rebuild sagelib whenever the public API of a dependency changes in a backwards-incompatible way. We devised "subslots" in Gentoo for the purpose of telling the package manager when it needs to rebuild. The documentation looks complicated but the basic idea is that we're copy/pasting the major component of the soname into the ebuild. When it changes, the package manager knows to rebuild the things that depend on it.

Rebuilding whenever any header of any dependency changes is an OK way to fake it, but it's going to trigger unnecessary sagelib rebuilds frequently.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,6 +1,6 @@
 Rebuilds should be triggered when headers of relevant packages change.
 
-In `setup.py` we have:
+In `setup.py` (after #29702, `sage_setup.command.sage_build_cython`) we have:
 
 ```
 # Do not put all, but only the most common libraries and their headers

[mkoeppe]

comment:4

Replying to @orlitzky:

What you really want to do here is rebuild sagelib whenever the public API of a dependency changes in a backwards-incompatible way. We devised "subslots" in Gentoo for the purpose of telling the package manager when it needs to rebuild. The documentation looks complicated

... indeed, from a quick look it seems there's a lot of Gentoo jargon there...

but the basic idea is that we're copy/pasting the major component of the soname into the ebuild. When it changes, the package manager knows to rebuild the things that depend on it.

Thanks for the pointer. It's probably a little bit more complicated for us because if I'm not mistaken, upstream packages are not always very careful with their sonames (which is then fixed in distributions)

[mkoeppe]

comment:5

The update of these paths, necessary because of the spkg-configure mechanism for GMP and NTL could use the environment variables set in sage-build-env-config.

[mkoeppe]

comment:6

I'll do this in #29847.

[mkoeppe]

comment:7

Replying to @mkoeppe:

I'll do this in #29847.

... or rather #29855.

[orlitzky]

comment:8

Replying to @mkoeppe:

Thanks for the pointer. It's probably a little bit more complicated for us because if I'm not mistaken, upstream packages are not always very careful with their sonames (which is then fixed in distributions)

In hindsight, it wouldn't really work for sage anyway. Since the (gentoo) developer makes up the subslot, he's able to change the subslot when the upstream API changes even if upstream doesn't change their soname. But rebuild-on-subslot-change only works for us because we know that all packages will be installed from the same repository. With sage, some packages may come from the system, and then you won't have access to that information. So rebuilding every time the headers change might be the best we can do.

[mkoeppe]

comment:9

See also #29791 comment 6

[mkoeppe]

comment:11

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[slel]

Description changed:

--- 
+++ 
@@ -12,6 +12,6 @@
                 "ntl":     [ os.path.join(SAGE_INC, 'NTL', 'config.h') ]
               }
 ```
-Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like gcc -M. This would be certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ).
+Except for this (which is much too limited and also needs updating in light of the spkg-configure mechanism), we do not have dependency tracking for sagelib. Neither of distutils, setuptools, cython seem to have a mechanism for generating dependencies like `gcc -M`. This would certainly be nice to have; in particular if it included dependencies on "system" headers and libraries, so that sagelib could automatically rebuild when system headers/libraries change (as in, most recently, ​https://groups.google.com/d/msg/sage-release/A53tGGsJNLg/uA01oUugAQAJ).
 
 (from #29411)

[dimpase]

comment:16

every sufficiently complicated Python module project will reimplement bits of make and autotools, in Python....