unix: rework terminfo database handling

Before, ncurses on Linux was built to look for the terminfo database in
a location that only existed in the build environment

After, it looks for the database in `/etc/terminfo`, `/lib/terminfo`,
and `/usr/share/terminfo`. These are the directory locations that
work on a superset of Debian and RedHat. This hopefully means that
terminfo database resolution "just works" in a lot of common Linux
environments.

As part of this, we add the terminfo database to the ncurses package
artifacts. Then we copy the terminfo database into
`install/share/terminfo` in the CPython distribution if a terminfo
database is present. macOS doesn't distribute a terminfo database
because the system terminfo database should be "good enough." Plus
we don't build ncurses on macOS distributions.

Closes #215.

Author: indygreg

cpython-unix/build-cpython.sh

@@ -884,6 +884,11 @@ if [ -d "${TOOLS_PATH}/deps/lib/tcl8" ]; then
     fi
 fi
 
+# Copy the terminfo database if present.
+if [ -d "${TOOLS_PATH}/deps/usr/share/terminfo" ]; then
+  cp -av ${TOOLS_PATH}/deps/usr/share/terminfo ${ROOT}/out/python/install/share/
+fi
+
 # config.c defines _PyImport_Inittab and extern references to modules, which
 # downstream consumers may want to strip. We bundle config.c and config.c.in so
 # a custom one can be produced downstream.

cpython-unix/build-ncurses.sh

@@ -20,7 +20,15 @@ if [[ -n "${CROSS_COMPILING}" && "${PYBUILD_PLATFORM}" != "macos" ]]; then
   echo "building host ncurses to provide modern tic for cross-compile"
 
   pushd ncurses-${NCURSES_VERSION}
-  CC="${HOST_CC}" ./configure --prefix=${TOOLS_PATH}/host --without-cxx --without-tests --without-manpages --enable-widec
+
+  CC="${HOST_CC}" ./configure \
+    --prefix=${TOOLS_PATH}/host \
+    --without-cxx \
+    --without-tests \
+    --without-manpages \
+    --enable-widec \
+    --disable-db-install \
+    --enable-symlinks
   make -j ${NUM_CPUS}
   make -j ${NUM_CPUS} install
 
@@ -38,6 +46,10 @@ pushd ncurses-${NCURSES_VERSION}
 # recognize the target architecture. We could fix this by overriding strip.
 # But we don't care about the installed binaries, so we simply disable
 # stripping of the binaries.
+# --enable-symlinks is needed to force use of symbolic links in the terminfo
+# database. By default hardlinks are used, which are wonky to tar up. Be sure
+# this is set on the host native `tic` build above, as it is the entity writing
+# symlinks!
 CONFIGURE_FLAGS="
     --build=${BUILD_TRIPLE}
     --host=${TARGET_TRIPLE}
@@ -46,7 +58,9 @@ CONFIGURE_FLAGS="
     --without-tests
     --without-manpages
     --disable-stripping
-    --enable-widec"
+    --enable-widec
+    --enable-symlinks
+    "
 
 # ncurses wants --with-build-cc when cross-compiling. But it insists on CC
 # and this value not being equal, even though using the same binary with
@@ -55,17 +69,48 @@ if [[ -n "${CROSS_COMPILING}" && "${PYBUILD_PLATFORM}" != "macos" ]]; then
   CONFIGURE_FLAGS="${CONFIGURE_FLAGS} --with-build-cc=$(which "${HOST_CC}")"
 fi
 
+# The terminfo database exists as a set of standalone files. The absolute
+# paths to these files need to be hardcoded into the binary at build time.
+#
+# Since our final distributions are "relocatable," the absolute path of the
+# terminfo database can't be known at build time: there needs to be something
+# that sniffs for well-known directories and attempts to locate it. Ideally
+# that could find the terminfo database that we ship!
+#
+# All is not lost, however.
+#
+# On macOS, the system terminfo database location is well known: /usr/share/terminfo.
+#
+# On Linux, common distributions tend to place the terminfo database in only a
+# few well-known locations. We define default search paths that overlap with
+# Debian and RedHat distros. This often results in at least a partially working
+# terminfo lookup in most Linux environments.
+#
+# configure appears to use --with-default-terminfo-dir for both a) where to
+# install the terminfo database to b) default TERMINFO value compiled into the
+# binary. So we provide a suitable runtime value and then move files at install
+# time.
+
 if [ "${PYBUILD_PLATFORM}" = "macos" ]; then
   CONFIGURE_FLAGS="${CONFIGURE_FLAGS}
     --datadir=/usr/share
     --sysconfdir=/etc
     --sharedstatedir=/usr/com
+    --with-default-terminfo-dir=/usr/share/terminfo
     --with-terminfo-dirs=/usr/share/terminfo
+  "
+else
+  CONFIGURE_FLAGS="${CONFIGURE_FLAGS}
+    --datadir=/tools/deps/usr/share
+    --sysconfdir=/tools/deps/etc
+    --sharedstatedir=/tools/deps/usr/com
     --with-default-terminfo-dir=/usr/share/terminfo
-    --disable-db-install
+    --with-terminfo-dirs=/etc/terminfo:/lib/terminfo:/usr/share/terminfo
   "
 fi
 
 CFLAGS="${EXTRA_TARGET_CFLAGS} -fPIC" CPPFLAGS="${EXTRA_TARGET_CFLAGS} -fPIC" LDFLAGS="${EXTRA_TARGET_LDFLAGS}" ./configure ${CONFIGURE_FLAGS}
 make -j ${NUM_CPUS}
 make -j ${NUM_CPUS} install DESTDIR=${ROOT}/out
+
+mv ${ROOT}/out/usr/share/terminfo ${ROOT}/out/tools/deps/usr/share/

docs/quirks.rst

@@ -13,6 +13,12 @@ If you attempt to run ``python`` and the backspace key doesn't
 erase characters or the arrow keys don't work as expected, this
 is because the executable can't find the *terminfo database*.
 
+A telltale sign of this is the Python REPL printing the following
+on startup::
+
+   Cannot read termcap database;
+   using dumb terminal settings.
+
 When you type a special key like the backspace key, this is
 registered as a key press. There is special software (typically
 ``readline`` or ``libedit``) that most interactive programs use
@@ -60,6 +66,20 @@ The macOS distributions built with this project should automatically
 use the terminfo database in ``/usr/share/terminfo``. Please file
 a bug report if the macOS distributions do not behave as expected.
 
+Starting in the first release after 20240107, the Linux distributions are
+configured to automatically use the terminfo database in ``/etc/terminfo``,
+``/lib/terminfo``, and ``/usr/share/terminfo``.
+
+Also starting in the first release after 20240107, the terminfo database
+is distributed in the ``share/terminfo`` directory (``../../share/terminfo``
+relative to the ``bin/python3`` executable) in Linux distributions. Note
+that ncurses and derived libraries don't know how to find this directory
+since they are configured to use absolute paths to the terminfo database
+and the absolute path of the Python distribution is obviously not known
+at build time! So actually using this bundled terminfo database will
+require custom code setting ``TERMINFO_DIRS`` before
+ncurses/libedit/readline are loaded.
+
 .. _quirk_tcl:
 
 Tcl/tk Support Files