Task ticket: Replace use of SAGE_ROOT by more specific environment variables

[mkoeppe]

#21501, #21534 replace $SAGE_ROOT/local by $SAGE_LOCAL (to make SAGE_LOCAL customizable via configure --prefix, #21479).

But there are also other direct uses of $SAGE_ROOT that should be replaced by something more specific, in particular in src/.

This is preparation for #21469 (Enable VPATH builds) and #21507 (Make sagelib a pip-installable Python source package, listed on PyPI).

See also #15105 - hardwired paths in src/sage.

The following list needs review.

find . -name local -prune -o -name _build -prune -o -name src/build -prune -o -name src/sage-7.4beta4 -prune -o -name .git -prune -o -type f -exec grep -nH -e SAGE_ROOT  {} +
nstall'"
...
./build/pkgs/autotools/spkg-src:5:if [ -z "$SAGE_ROOT" ]; then
...
./build/pkgs/autotools/spkg-write-makefile:13:if [ -z "$SAGE_ROOT" ]; then
...
./build/pkgs/configure/SPKG.txt:25:$SAGE_ROOT/bootstrap -s or the $SAGE_ROOT/src/bin/sage-update-version
./build/pkgs/d3js/spkg-src:5:[ -n "${SAGE_ROOT}" ] || SAGE_ROOT="$(pwd)/../../../"
./build/pkgs/d3js/spkg-src:33:mv "d3js-${ZIP_VERSION}.tar.gz" "${SAGE_ROOT}/upstream/"
./build/pkgs/database_gap/spkg-check:5:SMALL_GROUPS=`echo "SmallGroup(13,1); quit;" | $SAGE_ROOT/sage -gap -b -T | grep "13"`
./build/pkgs/database_stein_watkins_mini/spkg-install:4:. "$SAGE_ROOT/build/pkgs/database_stein_watkins/spkg-install"
./build/pkgs/ecl/spkg-install:77:cp "$SAGE_ROOT"/config/config.* src
./build/pkgs/ecl/spkg-src:16:if [ -z "$SAGE_ROOT" ] ; then
...
./build/pkgs/ecm/spkg-install:26:cp "$SAGE_ROOT"/config/config.* .
...
./build/pkgs/fflas_ffpack/spkg-install:23:cp "$SAGE_ROOT"/config/config.* build-aux
...
./build/pkgs/gap/spkg-install:55:cp "$SAGE_ROOT"/config/config.* cnf
./build/pkgs/gap3/spkg-src:19:if [ -z "$SAGE_ROOT" ] ; then
...
./build/pkgs/gcc/spkg-install:143:touch "$SAGE_ROOT/configure"
./build/pkgs/gf2x/spkg-install:23:cp "$SAGE_ROOT"/config/config.* config
./build/pkgs/givaro/spkg-install:38:cp "$SAGE_ROOT"/config/config.* build-aux
./build/pkgs/glpk/spkg-install:24:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/gmp/spkg-install:72:        # In some cases (see SAGE_ROOT/spkg/bin/sage-env), on Darwin,
./build/pkgs/gsl/SPKG.txt:99:   $SAGE_ROOT/spkg/standard/deps, and an explanation why GSL does *not*
./build/pkgs/jmol/spkg-install:2:if [ "$SAGE_ROOT" = "" ]; then
...
./build/pkgs/libfplll/spkg-src:11:SPKG_ROOT="$SAGE_ROOT/build/pkgs/libfplll"
./build/pkgs/linbox/spkg-install:25:cp "$SAGE_ROOT"/config/config.* build-aux
./build/pkgs/lrcalc/spkg-install:21:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/m4ri/SPKG.txt:42:   just use the version in the PATH (i.e., in SAGE_ROOT/spkg/bin)
./build/pkgs/m4rie/SPKG.txt:42:   $SAGE_ROOT/spkg/bin (in the $PATH).
./build/pkgs/mathjax/spkg-src:5:[ -n "${SAGE_ROOT}" ] || SAGE_ROOT="$(pwd)/../../../"
./build/pkgs/matplotlib/spkg-src:6:[ -n "${SAGE_ROOT}" ] || SAGE_ROOT="$(readlink -f $(dirname ${0})/../../../)"
...
./build/pkgs/maxima/spkg-install:44:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/mpfi/spkg-install:24:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/mpir/spkg-install:106:        # In some cases (see SAGE_ROOT/spkg/bin/sage-env), on Darwin,
./build/pkgs/ntl/spkg-src:11:SPKG_ROOT="$SAGE_ROOT/build/pkgs/ntl"
./build/pkgs/numpy/spkg-install:72:os.chdir(os.environ["SAGE_ROOT"])  # Import numpy from safe location
./build/pkgs/planarity/spkg-install:21:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/pycrypto/SPKG.txt:29: * config.patch: let pycrypto build when $SAGE_ROOT is in $PATH
./build/pkgs/python_igraph/spkg-src:3:# This script must be run from folder SAGE_ROOT. It modifies the file
./build/pkgs/python_igraph/spkg-src:8:if [ -z "$SAGE_ROOT" -o -z "$SAGE_DISTFILES" ]; then
...
./build/pkgs/qhull/patches/Makefile.patch:11:+DESTDIR = $(SAGE_ROOT)/local

(The above line is #21572.)

./build/pkgs/r/spkg-src:11:mv $SOURCE_TARBALL $SAGE_ROOT/upstream/$TARGET_TARBALL
./build/pkgs/sage_mode/spkg-install:56:    # Trick the shell into expanding $INSTALL_DIR and $SAGE_ROOT
./build/pkgs/sage_mode/SPKG.txt:50:(setq sage-command "$SAGE_ROOT/sage")
./build/pkgs/sagetex/SPKG.txt:46:The directory `$SAGE_ROOT/local/share/doc/sagetex` contains
./build/pkgs/sagetex/SPKG.txt:48:`$SAGE_ROOT/local/share/texmf/tex/latex/sagetex` for the source code and
./build/pkgs/scipoptsuite/SPKG.txt:36:scipoptsuite-VERSION.tgz in $SAGE_ROOT/upstream
./build/pkgs/sqlite/spkg-install:24:cp "$SAGE_ROOT"/config/config.* .
./build/pkgs/symmetrica/SPKG.txt:85: * make sure SAGE_ROOT is defined
./build/pkgs/threejs/spkg-src:8:[ -n "${SAGE_ROOT}" ] || SAGE_ROOT="$(pwd)/../../../"
...
./build/pkgs/zeromq/spkg-install:21:cp "$SAGE_ROOT"/config/config.* config
./build/sage_bootstrap/creator.py:21:from sage_bootstrap.env import SAGE_ROOT
./build/sage_bootstrap/creator.py:29:        self.path = os.path.join(SAGE_ROOT, 'build', 'pkgs', package_name)
./build/sage_bootstrap/env.py:8:* ``SAGE_ROOT``
...
./build/sage_bootstrap/package.py:22:from sage_bootstrap.env import SAGE_ROOT, SAGE_DISTFILES
...
./build/sage_bootstrap/updater.py:22:from sage_bootstrap.env import SAGE_ROOT
./build/test/test_package_cmdline.py:147:            env=dict(SAGE_ROOT=tmp)
./build/tox.ini:9:    SAGE_ROOT = {toxinidir}/..
./configure.ac:86:SAGE_ROOT=`pwd -P`
...
./Makefile:42:	export SAGE_ROOT=$$(pwd) && \
...
./README.md:215:         $SAGE_ROOT/local/share/doc/sage/html/
...
./sage:20:# Set SAGE_ROOT to the location of the sage install, i.e. the directory
....
./src/bin/sage-download-upstream:3:for pkg in $SAGE_ROOT/build/pkgs/*
./src/bin/sage-env:22:#  from the SAGE_ROOT directory.
...
./src/bin/sage-fix-pkg-checksums:7:# $SAGE_ROOT/upstream are checksummed.
./src/bin/sage-fixdoctests:131:    from sage.env import SAGE_ROOT
...
./src/bin/sage-list-packages:15:if "SAGE_ROOT" not in os.environ:
...
./src/bin/sage-location:7:SAGE_ROOT     = os.path.realpath(os.environ['SAGE_ROOT'])
...
./src/bin/sage-notebook:13:from sage.env import SAGE_ROOT
./src/bin/sage-README-osx.txt:89:preferences.  You can also drag a sage folder (SAGE_ROOT) onto
...
./src/bin/sage-rebase.bat:5:rem Rebase all dlls in the SAGE_ROOT directory (and its subdirectories),
...
./src/bin/sage-rebase.sh:6:# Rebase all dlls in the SAGE_ROOT directory (and its subdirectories),
...
./src/bin/sage-rebaseall.bat:6:rem Rebase all dlls in the SAGE_ROOT directory (and its subdirectories),
...
./src/bin/sage-rebaseall.sh:8:# Rebase all dlls in the SAGE_ROOT directory (and its subdirectories)
...
./src/bin/sage-rsyncdist:67:# Run this script from SAGE_ROOT
...
./src/bin/sage-sdist:26:if [ -z "$SAGE_ROOT" ]; then
...
./src/bin/sage-starts:3:# Run this test from SAGE_ROOT.
...
./src/bin/sage-sws2rst:151:file in $SAGE_ROOT/src/doc to include your file, or
./src/bin/sage-update-version:24:if [ -z "$SAGE_ROOT" ]; then
...
./src/bin/sage-upgrade:3:# This script must be run from $SAGE_ROOT and without having sourced
./src/bin/testcc.sh:68:if [ -z "$SAGE_ROOT" ]; then
...
./src/bin/testcflags.sh:79:if [ -z "$SAGE_ROOT" ]; then
...
./src/bin/testcxx.sh:68:if [ -z "$SAGE_ROOT" ]; then
...
./src/doc/common/themes/sage/static/thebe-sage.js:9: * installed in ${SAGE_ROOT}/local/share/thebe/thebe.js by Sage's thebe package.
./src/doc/.... (many)
...
./src/mac-app/AppController.m:427:    // If the file is a directory, see if it's SAGE_ROOT
./src/mac-app/Makefile:16:ifdef SAGE_ROOT
...
./src/sage/all.py:76:from sage.env import SAGE_ROOT, SAGE_SRC, SAGE_DOC_SRC, SAGE_LOCAL, DOT_SAGE, SAGE_ENV
./src/sage/calculus/desolvers.py:1623:                          os.path.join('$SAGE_ROOT','local','lib','libTIDES.a') + ' $LDFLAGS '
...
./src/sage/doctest/control.py:635:        from sage.env import SAGE_SRC, SAGE_DOC_SRC, SAGE_ROOT
...
./src/sage/env.py:92:_add_variable_or_fallback('SAGE_ROOT',       None)
...
./src/sage/misc/citation.pyx:4:from sage.env import SAGE_ROOT
./src/sage/misc/citation.pyx:88:    strings = [a[0].replace(SAGE_ROOT, "") + " " + a[2] for a in stats.stats.keys()]
./src/sage/misc/copying.py:7:from sage.env import SAGE_ROOT
./src/sage/misc/copying.py:17:        return open(os.path.join(SAGE_ROOT,'COPYING.txt')).read()

The above is #21571.

./src/sage/misc/cython_c.pyx:29:      ``$SAGE_ROOT`` is expanded, but other environment variables
./src/sage/misc/edit_module.py:292:    'SAGE_ROOT/src'.
./src/sage/misc/inline_fortran.py:70:            sage: os.chdir(SAGE_ROOT)
./src/sage/misc/inline_fortran.py:75:            sage: os.getcwd() == SAGE_ROOT
./src/sage/misc/misc.py:111:        sage: sage_executable = os.path.join(SAGE_ROOT, 'sage')
./src/sage/misc/package.py:46:from sage.env import SAGE_ROOT
./src/sage/misc/package.py:62:SAGE_PKGS = os.path.join(SAGE_ROOT, "build", "pkgs")
./src/sage/misc/package.py:366:    package has a directory in ``SAGE_ROOT/build/pkgs/``, then
./src/sage/misc/sage_ostools.py:27:        sage: have_program('sage', path=SAGE_ROOT)
./src/sage/misc/sage_ostools.py:29:        sage: have_program('ls', path=SAGE_ROOT)
./src/sage/repl/ipython_kernel/install.py:13:    SAGE_ROOT, SAGE_DOC, SAGE_LOCAL, SAGE_EXTCODE,
./src/sage/repl/ipython_kernel/install.py:154:            os.path.join(SAGE_ROOT, 'sage'),
./src/sage/structure/sage_object.pyx:1376:    that and doctest Sage, you'll find that the ``SAGE_ROOT``/tmp/
./src/sage/structure/sage_object.pyx:1386:      ``SAGE_ROOT/tmp/pickle_jar``
./src/sage/structure/sage_object.pyx:1416:        dir = os.environ['SAGE_ROOT'] + '/tmp/pickle_jar/'
./src/sage/structure/sage_object.pyx:1532:        copying the files from ``SAGE_ROOT/tmp/pickle_jar*`` into the standard pickle

Depends on #21571
Depends on #21572
Depends on #15105

CC: @jdemeyer @embray @kiwifb @jhpalmieri @tscrim

Component: build

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/21591

[mkoeppe]

Description changed:

--- 
+++ 
@@ -3,6 +3,8 @@
 But there are also other direct uses of `$SAGE_ROOT` that should be replaced by something more specific, in particular in `src/`.
 
 This is preparation for #21469 (Enable VPATH builds) and #21507 (Make sagelib a pip-installable Python source package, listed on PyPI).
+
+See also #15105 - hardwired paths in `src/sage`.
 
 The following list needs review.
 

[embray]

comment:2

Thanks--yeah I've encountered a ton of this. Likewise there are misuses of $SAGE_LOCAL scattered about when there are other, more specific variables that could be used. I mostly fix them when I see them, but haven't considered doing a comprehensive cleanup.

[mkoeppe]

Description changed:

--- 
+++ 
@@ -58,6 +58,10 @@
 ./build/pkgs/python_igraph/spkg-src:8:if [ -z "$SAGE_ROOT" -o -z "$SAGE_DISTFILES" ]; then
 ...
 ./build/pkgs/qhull/patches/Makefile.patch:11:+DESTDIR = $(SAGE_ROOT)/local
+```
+(The above line is #21572.)
+
+```
 ./build/pkgs/r/spkg-src:11:mv $SOURCE_TARBALL $SAGE_ROOT/upstream/$TARGET_TARBALL
 ./build/pkgs/sage_mode/spkg-install:56:    # Trick the shell into expanding $INSTALL_DIR and $SAGE_ROOT
 ./build/pkgs/sage_mode/SPKG.txt:50:(setq sage-command "$SAGE_ROOT/sage")

[embray]

comment:4

Has anyone done any work on this yet? I'm in the mood for a (somewhat) mindless and tedious task.

[kiwifb]

comment:5

Be my guest! I historically reduced the use of SAGE_ROOT in the build system and at runtime. Most of the usage in src/sage, with the exception of env.py is replaced or removed (package.py) one way or another in sage-on-gentoo.

[embray]

comment:6

fbissey: Will improvements on this front then cut down on the amount of contortions sage-on-gentoo needs to go through?

[kiwifb]

comment:7

If done right, yes. Although looking at the list of files there are a variety of issues. I would move package.py completely to sage_setup so it is never called at runtime. I also moved repl/ipython_kernel/install.py to sage_setup as similarly you should only use it at build/install time. I have touched the issue with Jeroen before but he wanted to wait to see if the way to install jupyter kernel would get "formalised".

In a lot of other instances SAGE_ROOT is replaced by SAGE_LIB.

[mkoeppe]

comment:8

Replying to @embray:

Has anyone done any work on this yet? I'm in the mood for a (somewhat) mindless and tedious task.

Please go ahead!

[embray]

comment:9

It looks like #15105 got a lot of this done already. I'll see if I can help there first.

Or more specifically, the work should probably be done on top of #15105 so as not to conflict with it.

[embray]

comment:10

For picklejar, I know it's mostly a utility for testing, but SAGE_ROOT/tmp hardly seems the appropriate place for it by default.

I would think instead the current working directory is appropriate in general--for running the tests I would just put it in SAGE_TMP.

[embray]

Dependencies: #21571

[embray]

comment:11

I think #21571 should be considered a dependency, since it's definitely a piece of this puzzle.

[mkoeppe]

Changed dependencies from #21571 to #21571, #21572, #15105

[mkoeppe]

Description changed:

--- 
+++ 
@@ -144,6 +144,10 @@
 ./src/sage/misc/citation.pyx:88:    strings = [a[0].replace(SAGE_ROOT, "") + " " + a[2] for a in stats.stats.keys()]
 ./src/sage/misc/copying.py:7:from sage.env import SAGE_ROOT
 ./src/sage/misc/copying.py:17:        return open(os.path.join(SAGE_ROOT,'COPYING.txt')).read()
+```
+The above is #21571.
+
+```
 ./src/sage/misc/cython_c.pyx:29:      ``$SAGE_ROOT`` is expanded, but other environment variables
 ./src/sage/misc/edit_module.py:292:    'SAGE_ROOT/src'.
 ./src/sage/misc/inline_fortran.py:70:            sage: os.chdir(SAGE_ROOT)

[mkoeppe]

comment:13

See also: #25150