Meta-ticket: Make sage (the distribution) behave like a standard autotools package, to the extent possible

[mkoeppe]

The goal of this ticket is to be able to explain to experienced Unix users what sage-the-distribution is, as follows:

If you download sage-the-distribution and do

./configure --prefix=/SOMEWHERE && make 

then that's the same as doing a sequence of about 100 times:

wget SOMEPACKAGE.tar.gz && tar xf SOMEPACKAGE.tar.gz && (cd SOMEPACKAGE \
&& ./configure --prefix=/SOMEWHERE && make && make install)

(and about a 50 times pip install SOMEPACKAGE).
sage-the-distribution has figured out the right order of installing these packages, tricky configure options so that everything works, and on top has fixes for various outdated/handwritten/missing build systems of various packages.

Thus it is beyond the scope of this ticket:

• to separate make and make install. Our make install is a no-op. ANY discussion of this needs to go to ticket Long-term task ticket: Support "make install" of the Sage distribution #21495, not here.

We will implement this goal without sacrificing any of the traditional convenience features that Sage-the-distribution has provided for the casual user (such as sage -i for installing packages; and that ./configure is an optional step of the installation process).

We have a separate task ticket for the following:

• Meta-ticket: Make sagelib a pip-installable Python source package, listed on PyPI #21507: Make sagelib (sage-the-Python-library) a normal Python package, installable by setup.py, pip, etc. and eventually even via PyPI. We will NOT turn sagelib to an automake package (which was proposed in Transition to build system for sage (the library) #14807).
  (There will be some interaction with some of the steps of that ticket.)

Included on this ticket are the following steps.

Implement standard features expected of an autotools build system.

• Choosing the installation hierarchy (configure --prefix=SAGE_LOCAL). Right now it is the subdirectory local of SAGE_ROOT.
  • Allow SAGE_LOCAL to be customized #21501: Allow SAGE_LOCAL to be customized
  • Allow SAGE_LOCAL to be customized - follow-up #21534: Allow SAGE_LOCAL to be customized - follow-up
  • ./configure --prefix=SAGE_LOCAL #21479: ./configure --prefix=SAGE_LOCAL
• By allowing the user to choose the installation hierarchy, there are new requirements. What is installed there should run without requiring environment variables to be set. It should not refer to the environment variables SAGE_LOCAL, or SAGE_ROOT (or SAGE_SRC_ROOT). An exception could perhaps be made for the latter for "debugging" or "source inspection" facilities:
  • package autotools: Don't depend on $SAGE_LOCAL #21525: package autotools: Don't depend on $SAGE_LOCAL
  • Improve Cython debugging #21509: Install cython_debug somewhere in SAGE_LOCAL
• make V=0 should silence the build #21539: make V=0 should silence the build
• make distclean: Don't delete $SAGE_ROOT/local #21775: make distclean: Don't delete the install hierarchy (SAGE_LOCAL)
• Task ticket: Replace use of SAGE_ROOT by more specific environment variables #21591: Replace use of SAGE_ROOT by more specific environment variables
• Enable VPATH builds (several independent build trees connected to one source tree) #21469: Enable VPATH builds (SAGE_SRC_ROOT/configure --srcdir=SAGE_SRC_ROOT)
• ./configure CC=/path/to/gcc ... #22646: ./configure CC=/path/to/gcc ...
• Add --enable-OPTIONALSPKG options to configure #28095: Add --enable-OPTIONALSPKG options to configure (see also earlier ./configure --with-packages=... #21538)
• Reimplement sage -i SPKG for optional/experimental packages as configure --enable-SPKG && make build #29113: Reimplement sage -i SPKG for optional/experimental packages as configure --enable-SPKG && make build (follow-up: Remove ./configure --enable-SPKG options for pip packages (for which this is not implemented) #29626)
• Allow users to make make V=0 the default by using ./configure --enable-silent-rules #21589: Make make V=0 the default by ./configure --enable-silent-rules
• Various other configure options, to replace use of environment variables that influence the build.

Clean up parts of the build system to make it more standard. This is to make it straightforward for developers familiar with the autotools system to contribute to sage.

• Create SAGE_LOCAL directory hierarchy during "make", not "configure" #21532: Create SAGE_LOCAL directory hierarchy during make, not configure
• configure.ac: write build/make/Makefile within an AC_CONFIG_FILE, not during main configure #21524: configure.ac: write build/make/Makefile within an AC_CONFIG_COMMANDS, not during main configure

Make the separation between sage-the-distribution and sagelib (sage-the-Python-library) clearer. This will be beneficial for distributions such as Debian etc.

• Install src/bin scripts by sagelib's setup.py, not make #21559: Move sage-the-distribution scripts from src/bin to build/bin
• Add src/README.txt and build/README.txt #21565: Add src/README.txt and build/README.txt

Following are workarounds to enable root-owned installation hierarchies (prefix).

• If $SAGE_SUDO is set, use it whenever we do "make install" of a package. #21537: If $SAGE_SUDO is set, use it whenever we do make install of a package. Subtickets:
  • Support splitting spkg install into spkg-build and spkg-install (for SAGE_SUDO) #21726: Support splitting spkg install into spkg-build and spkg-install (for SAGE_SUDO)
• documentation: Recommend GNU stow to people who want to install to /usr/local #21536: documentation: Recommend GNU stow to people who want to install to /usr/local

See also:

• Meta-meta-ticket: Build, packaging, testing improvements #29133 META-META-TICKET: Build, packaging, testing improvements

CC: @vbraun @jdemeyer @kiwifb @embray @dimpase @williamstein @mezzarobba @tobihan @timokau @jhpalmieri @seblabbe

Component: build

Author: Matthias Koeppe

Issue created by migration from https://trac.sagemath.org/ticket/21566

[mezzarobba]

comment:3

Do you intend ./configure to detect already installed dependencies so that make skips installing them? (Eventually? as part of this task?) For example, is the idea that ./configure would finish by saying something like “the following dependencies are missing: ... please install them, or type make and I'll install my own copy”? (This wouldn't be quite standard, but close enough, while staying mostly compatible with the installation procedure of sage-the-distribution.)

Or would sage-the-distribution still always have its private version of everything, so that users would have to choose between installing it and installing sagelib directly (e.g., via pip)?

[embray]

comment:4

(Sorry, the comment that was here previously was meant to go on #21507 -- I have too many tabs open)

[mkoeppe]

comment:5

Replying to @mezzarobba:

Do you intend ./configure to detect already installed dependencies so that make skips installing them? (Eventually? as part of this task?)

That is beyond the scope of this task ticket. But I would be interested in a follow-up ticket to this effect.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -35,6 +35,7 @@
   - #21525: package `autotools`: Don't depend on `$SAGE_LOCAL`
   - #21509: Install cython_debug somewhere in SAGE_LOCAL
 - #21539: `make V=0` should silence the build
+- #21589: Make `make V=0` the default by `./configure --enable-silent-rules`
 - #21469: Enable VPATH builds (`SAGE_SRC_ROOT/configure --srcdir=SAGE_SRC_ROOT`)
 - #21538: `./configure --with-packages=...`
 - Various other `configure` options, to replace use of environment variables that influence the build.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -31,7 +31,7 @@
   - #21501: Allow SAGE_LOCAL to be customized
   - #21534: Allow SAGE_LOCAL to be customized - follow-up
   - #21479: ./configure --prefix=SAGE_LOCAL
-- By allowing the user to choose the installation hierarchy, there are new requirements. What is installed there should run without requiring environment variables to be set. It should not refer to `SAGE_ROOT` (or `SAGE_SRC_ROOT`), except perhaps for "debugging" or "source inspection" facilities:
+- By allowing the user to choose the installation hierarchy, there are new requirements. What is installed there should run without requiring environment variables to be set. It should not refer to the environment variables `SAGE_LOCAL`, or `SAGE_ROOT` (or `SAGE_SRC_ROOT`). An exception could perhaps be made for the latter for "debugging" or "source inspection" facilities:
   - #21525: package `autotools`: Don't depend on `$SAGE_LOCAL`
   - #21509: Install cython_debug somewhere in SAGE_LOCAL
 - #21539: `make V=0` should silence the build

[mkoeppe]

Description changed:

--- 
+++ 
@@ -36,6 +36,7 @@
   - #21509: Install cython_debug somewhere in SAGE_LOCAL
 - #21539: `make V=0` should silence the build
 - #21589: Make `make V=0` the default by `./configure --enable-silent-rules`
+- #21591: Replace use of `SAGE_ROOT` by more specific environment variables
 - #21469: Enable VPATH builds (`SAGE_SRC_ROOT/configure --srcdir=SAGE_SRC_ROOT`)
 - #21538: `./configure --with-packages=...`
 - Various other `configure` options, to replace use of environment variables that influence the build.

[embray]

comment:9

Will this also fix how make actually runs configure for you, and that configure gets invoked even if you run make clean targets?

[mkoeppe]

comment:10

Replying to @embray:

Will this also fix how make actually runs configure for you,

No, probably not. For this ticket, I want to keep configure an optional step of the installation process. (I've updated the description.)

and that configure gets invoked even if you run make clean targets?

Not sure about this one. To keep this ticket manageable, I want to change the top-level Makefile as little as possible. But please do open a ticket that describes make clean issues that ought to be addressed.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -17,7 +17,7 @@
 Thus it is beyond the scope of this ticket:
 - to separate `make` and `make install`. Our `make install` is a no-op. ANY discussion of this needs to go to ticket #21495, not here.
 
-We will implement this goal without sacrificing any of the traditional convenience features that Sage-the-distribution has provided for the casual user (such as `sage -i` for installing packages).
+We will implement this goal without sacrificing any of the traditional convenience features that Sage-the-distribution has provided for the casual user (such as `sage -i` for installing packages; and that `./configure` is an *optional* step of the installation process).
 
 We have a separate task ticket for the following:
 - #21507: Make sagelib (sage-the-Python-library) a normal Python package, installable by `setup.py`, `pip`, etc. and eventually even via PyPI.  We will NOT turn sagelib to an automake package (which was proposed in #14807). 

[embray]

comment:11

Replying to @mkoeppe:

Replying to @embray:

Will this also fix how make actually runs configure for you,

No, probably not. For this ticket, I want to keep configure an optional step of the installation process. (I've updated the description.)

I guess it would be fine to make it an optional step. But I don't think it should be a mandatory step for all make targets, and currently it is, or at least seems to be. For example if I run make maintainer-clean twice in a row, the second time will run configure only to delete its output.

This is especially annoying on Cygwin where configure is extremely slow.

[jdemeyer]

comment:12

Replying to @embray:

Will this also fix how make actually runs configure for you, and that configure gets invoked even if you run make clean targets?

In standard autotools packages, the Makefile gets created by configure, so you wouldn't be able to run make clean either without configure.

[embray]

comment:13

True--in that case it should (hopefully) be a moot point. This is currently a huge annoyance for me.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -51,4 +51,5 @@
 
 Following are workarounds to enable root-owned installation hierarchies (`prefix`).
 - #21537: If `$SAGE_SUDO` is set, use it whenever we do `make install` of a package
+- #21726: Support splitting spkg install into `spkg-build` and `spkg-install` (for SAGE_SUDO)
 - #21536: documentation: Recommend GNU stow to people who want to install to `/usr/local`

[mkoeppe]

Description changed:

--- 
+++ 
@@ -35,6 +35,7 @@
   - #21525: package `autotools`: Don't depend on `$SAGE_LOCAL`
   - #21509: Install cython_debug somewhere in SAGE_LOCAL
 - #21539: `make V=0` should silence the build
+- #21775: make distclean: Don't delete the install hierarchy (SAGE_LOCAL)
 - #21589: Make `make V=0` the default by `./configure --enable-silent-rules`
 - #21591: Replace use of `SAGE_ROOT` by more specific environment variables
 - #21469: Enable VPATH builds (`SAGE_SRC_ROOT/configure --srcdir=SAGE_SRC_ROOT`)

[mkoeppe]

Description changed:

--- 
+++ 
@@ -51,6 +51,6 @@
 - #21565: Add src/README.txt and build/README.txt
 
 Following are workarounds to enable root-owned installation hierarchies (`prefix`).
-- #21537: If `$SAGE_SUDO` is set, use it whenever we do `make install` of a package
-- #21726: Support splitting spkg install into `spkg-build` and `spkg-install` (for SAGE_SUDO)
+- #21537: If `$SAGE_SUDO` is set, use it whenever we do `make install` of a package. Subtickets:
+  - #21726: Support splitting spkg install into `spkg-build` and `spkg-install` (for SAGE_SUDO)
 - #21536: documentation: Recommend GNU stow to people who want to install to `/usr/local`