make V=0 should silence the build

[mkoeppe]

Building sage outputs a lot. Especially in highly parallelized builds, the output is not very useful.

This ticket implements an option to silence the build, using make argument (or environment variable) V=0. (This mimics the interface of Automake silent rules; so this is one step towards #21566):

$ make -j8 V=0 autotools
make -j8 build/make/Makefile
make[1]: warning: -jN forced in submake: disabling jobserver mode.
make[1]: `build/make/Makefile' is up to date.
*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***
... long list of environment variables elided -- this ticket does nothing about that ...
***********************************************
make[1]: warning: -jN forced in submake: disabling jobserver mode.
Running 'sage-spkg xz-5.2.2', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/xz-5.2.2.log...
Running 'sage-spkg xz-5.2.2', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/xz-5.2.2.log... done
Running 'sage-spkg autotools-20141105', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/autotools-20141105.log...
=========================== WARNING ===========================
You are about to download and install the experimental package
autotools-20141105.  This probably won't work at all for you! There
is no guarantee that it will build correctly, or behave as 
expected. Use at your own risk!
===============================================================
Are you sure you want to continue [Y/n]? 
OK, installing autotools-20141105 now.
Running 'sage-spkg autotools-20141105', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/autotools-20141105.log... exit status 1
make[1]: *** [/Users/mkoeppe/cvs/sage/local/var/lib/sage/installed/autotools-20141105] Error 1

Note that, though all output is redirected, the interactive questions (#20884, #21082) still go through to the terminal. This is achieved by using /dev/tty.

The patch also unconditionally silences a few make recipe lines involving sage-logger.

Follow-up ticket: #21589

CC: @embray @jdemeyer @vbraun @videlec @dimpase @nexttime @kcrisman @novoselt @kiwifb

Component: build

Author: Matthias Koeppe

Branch: 740ecfb

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/21539

[mkoeppe]

Description changed:

--- 
+++ 
@@ -15,4 +15,4 @@
 `make V=0` 
 would request the silent mode. `./configure --enable-silent-rules` would make this the default.
 
-
+Of course, in silent mode it's imperative that there's no interactivity. So we need the "proper solution for warnings about experimental packages" (#21082)

[mkoeppe]

Branch: u/mkoeppe/make_v_0_should_silence_the_build

[mkoeppe]

New commits:

75223db	Silence some make rules
0361466	Use /dev/tty for interaction
2322391	If V=0, redirect (not tee) to logfile

[mkoeppe]

Commit: 2322391

[mkoeppe]

Author: Matthias Koeppe

[mkoeppe]

Description changed:

--- 
+++ 
@@ -1,18 +1,34 @@
 Building sage outputs a lot. Especially in highly parallelized builds, the output is not very useful. 
 
-We should have an option to silence the build. 
-Then it should simply say:
+This ticket implements an option to silence the build, using `make` argument (or environment variable) `V=0`. (This mimics the interface of [Automake silent rules](https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rules.html#Automake-Silent-Rules); so this is one step towards #21566):
 
 ```
-[cvxopt-1.1.8.p1] Logging to '/Users/mkoeppe/cvs/sage/logs/pkgs/cvxopt-1.1.8.p1.log'
-[cvxopt-1.1.8.p1] Using cached file /Users/mkoeppe/cvs/sage/upstream/cvxopt-1.1.8.tar.gz
-[cvxopt-1.1.8.p1] Setting up build directory for cvxopt-1.1.8.p1
-[cvxopt-1.1.8.p1] Building and installing
-[cvxopt-1.1.8.p1] Installation completed
+$ make -j8 V=0 autotools
+make -j8 build/make/Makefile
+make[1]: warning: -jN forced in submake: disabling jobserver mode.
+make[1]: `build/make/Makefile' is up to date.
+*** ALL ENVIRONMENT VARIABLES BEFORE BUILD: ***
+... long list of environment variables elided -- this ticket does nothing about that ...
+***********************************************
+make[1]: warning: -jN forced in submake: disabling jobserver mode.
+Running 'sage-spkg xz-5.2.2', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/xz-5.2.2.log...
+Running 'sage-spkg xz-5.2.2', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/xz-5.2.2.log... done
+Running 'sage-spkg autotools-20141105', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/autotools-20141105.log...
+=========================== WARNING ===========================
+You are about to download and install the experimental package
+autotools-20141105.  This probably won't work at all for you! There
+is no guarantee that it will build correctly, or behave as 
+expected. Use at your own risk!
+===============================================================
+Are you sure you want to continue [Y/n]? 
+OK, installing autotools-20141105 now.
+Running 'sage-spkg autotools-20141105', output appears in /Users/mkoeppe/cvs/sage/logs/pkgs/autotools-20141105.log... exit status 1
+make[1]: *** [/Users/mkoeppe/cvs/sage/local/var/lib/sage/installed/autotools-20141105] Error 1
 ```
 
-A possible interface is to mimic [Automake silent rules](https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rules.html#Automake-Silent-Rules):
-`make V=0` 
-would request the silent mode. `./configure --enable-silent-rules` would make this the default.
+Note that, though all output is redirected, the interactive questions (#20884, #21082) still go through to the terminal. This is achieved by using `/dev/tty`.
 
-Of course, in silent mode it's imperative that there's no interactivity. So we need the "proper solution for warnings about experimental packages" (#21082)
+The patch also unconditionally silences a few make recipe lines involving `sage-logger`.
+
+Follow-up ticket: #21589
+

[mkoeppe]

comment:6

Needs review.

[jdemeyer]

comment:7

I think the /dev/tty thing should be a separate ticket, since it's very specific and might require a specific reviewer who knows what it is about.

[jdemeyer]

comment:8

I really dislike @ in Makefiles (unless the rule is truly trivial). It has happened to me several times that I was debugging some build issue and the @ was hiding important stuff.

[mkoeppe]

comment:9

Replying to @jdemeyer:

I think the /dev/tty thing should be a separate ticket, since it's very specific and might require a specific reviewer who knows what it is about.

If Erik doesn't mind, I could repurpose #21082 for that.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

04aa4f7	Merge tag '7.4.beta6' into t/21539/make_v_0_should_silence_the_build
850eb86	Only silence rules when V=0

[sagetrac-git]

Changed commit from 2322391 to 850eb86

[mkoeppe]

comment:11

Replying to @jdemeyer:

I really dislike @ in Makefiles (unless the rule is truly trivial). It has happened to me several times that I was debugging some build issue and the @ was hiding important stuff.

OK, I've changed it so this is only done in V=0 mode.

[jhpalmieri]

comment:12

Two comments:

• in the past (see decrease the size of SAGE_ROOT/install.log #12729, for example), there were objections (by Jeroen, and maybe others) to not producing the full install.log file as is done now: this can be useful for debugging race conditions.

• I think the output hard to read. I would like something like the proposed output at decrease the size of SAGE_ROOT/install.log #12729 better. That is, change

Running 'sage-spkg mpc-1.0.3.p0', output appears in /path/to/SAGE_ROOT/log/pkgs/mpc-1.0.3.p0.log ...
Running 'sage-spkg mpc-1.0.3.p0', output appears in /path/to/SAGE_ROOT/log/pkgs/mpc-1.0.3.p0.log ... done

to something like

[mpc-1.0.3.p0] installing. Log file: logs/pkgs/mpc-1.0.3.p0.log
  [mpc-1.0.3.p0] installed successfully.

The path to the log file should be given relative to SAGE_ROOT, which shortens the message, making it less likely that the line wraps, since line-wrapping is part of what makes the output hard to read.

[jhpalmieri]

comment:13

For the output, I'm thinking of something like this change (although I cheat in constructing the shortened path for the logfile here -- there should be a more robust solution):

diff --git a/build/bin/sage-logger b/build/bin/sage-logger
index c6afae1..de6cff8 100755
--- a/build/bin/sage-logger
+++ b/build/bin/sage-logger
@@ -57,15 +57,15 @@ mkdir -p "$logdir"
 if [[ "$V" = 0 && $use_prefix = true ]]; then
     # Silent build.
     # Similar to https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rules.html#Automake-Silent-Rules
-    echo "Running '$cmd', output appears in $logfile..."
+    echo "[$logname] installing. Log file: logs/pkg/$logname"
     # Use verbose mode for output to logfiles.
     export V=1
     ( exec> $logfile 2>&1 ; eval "$cmd" )
     status=$?
     if [[ $status != 0 ]]; then
-       echo "Running '$cmd', output appears in $logfile... exit status $status"
+       echo "  [$logname] error installing, exit status $status. Log file: $logfile"
     else
-       echo "Running '$cmd', output appears in $logfile... done"
+       echo "  [$logname] successfully installed"
     fi
     exit $status
 else