GH-103065, GH-106704, GH-105253: Provide a Tools/wasm/wasi.py script to simplify doing a WASI build (GH-112473)

Author: brettcannon

.devcontainer/Dockerfile

@@ -6,7 +6,7 @@ ENV WASI_SDK_VERSION=20
 ENV WASI_SDK_PATH=/opt/wasi-sdk
 
 ENV WASMTIME_HOME=/opt/wasmtime
-ENV WASMTIME_VERSION=9.0.1
+ENV WASMTIME_VERSION=14.0.4
 ENV WASMTIME_CPU_ARCH=x86_64
 
 RUN dnf -y --nodocs --setopt=install_weak_deps=False install /usr/bin/{blurb,clang,curl,git,ln,tar,xz} 'dnf-command(builddep)' && \

.gitignore

@@ -125,6 +125,7 @@ Tools/unicode/data/
 /config.status.lineno
 # hendrikmuhs/ccache-action@v1
 /.ccache
+/cross-build/
 /platform
 /profile-clean-stamp
 /profile-run-stamp

Makefile.pre.in

@@ -2732,6 +2732,7 @@ clobber: clean
 	-rm -rf build platform
 	-rm -rf $(PYTHONFRAMEWORKDIR)
 	-rm -f python-config.py python-config
+	-rm -rf cross-build
 
 # Make things extra clean, before making a distribution:
 # remove all generated files, even Makefile[.pre]

Misc/NEWS.d/next/Build/2023-11-27-13-55-47.gh-issue-103065.o72OiA.rst

@@ -0,0 +1 @@
+Introduce ``Tools/wasm/wasi.py`` to simplify doing a WASI build.

Tools/wasm/README.md

@@ -298,100 +298,66 @@ AddType application/wasm wasm
 
 ## WASI (wasm32-wasi)
 
-WASI builds require the [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 16.0+.
-See `.devcontainer/Dockerfile` for an example of how to download and
-install the WASI SDK.
+**NOTE**: The instructions below assume a Unix-based OS due to cross-compilation for CPython being set up for `./configure`.
 
-### Build
+### Prerequisites
+
+Developing for WASI requires two things to be installed:
+
+1. The [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 16.0+
+   (see `.devcontainer/Dockerfile` for an example of how to download and install the WASI SDK)
+2. A WASI host/runtime ([wasmtime](https://wasmtime.dev) 14+ is recommended and what the instructions below assume)
 
-The script ``wasi-env`` sets necessary compiler and linker flags as well as
-``pkg-config`` overrides. The script assumes that WASI-SDK is installed in
-``/opt/wasi-sdk`` or ``$WASI_SDK_PATH``.
 
-There are two scripts you can use to do a WASI build from a source checkout. You can either use:
+### Building
 
+Building for WASI requires doing a cross-build where you have a "build" Python to help produce a WASI build of CPython (technically it's a "host x host" cross-build because the build Python is also the target Python while the host build is the WASI build; yes, it's confusing terminology). In the end you should have a build Python in `cross-build/build` and a WASI build in `cross-build/wasm32-wasi`.
+
+The easiest way to do a build is to use the `wasi.py` script. You can either have it perform the entire build process from start to finish in one step, or you can do it in discrete steps that mirror running `configure` and `make` for each of the two builds of Python you end up producing (which are beneficial when you only need to do a specific step after getting a complete build, e.g. editing some code and you just need to run `make` for the WASI build).
+
+The discrete steps are:
 ```shell
-./Tools/wasm/wasm_build.py wasi build
+python Tools/wasm/wasi.py configure-build-python
+python Tools/wasm/wasi.py make-build-python
+python Tools/wasm/wasi.py configure-host
+python Tools/wasm/wasi.py make-host
 ```
 
-or:
+To do it in a single command, run:
 ```shell
-./Tools/wasm/build_wasi.sh
+python Tools/wasm/wasi.py build
 ```
 
-The commands are equivalent to the following steps:
-
-- Make sure `Modules/Setup.local` exists
-- Make sure the necessary build tools are installed:
-  - [WASI SDK](https://github.com/WebAssembly/wasi-sdk) (which ships with `clang`)
-  - `make`
-  - `pkg-config` (on Linux)
-- Create the build Python
-  - `mkdir -p builddir/build`
-  - `pushd builddir/build`
-  - Get the build platform
-    - Python: `sysconfig.get_config_var("BUILD_GNU_TYPE")`
-    - Shell: `../../config.guess`
-  - `../../configure -C`
-  - `make all`
-  - ```PYTHON_VERSION=`./python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")'` ```
-  - `popd`
-- Create the host/WASI Python
-  - `mkdir builddir/wasi`
-  - `pushd builddir/wasi`
-  - `../../Tools/wasm/wasi-env ../../configure -C --host=wasm32-unknown-wasi --build=$(../../config.guess) --with-build-python=../build/python`
-    - `CONFIG_SITE=../../Tools/wasm/config.site-wasm32-wasi`
-    - `HOSTRUNNER="wasmtime run --mapdir /::$(dirname $(dirname $(pwd))) --env PYTHONPATH=/builddir/wasi/build/lib.wasi-wasm32-$PYTHON_VERSION $(pwd)/python.wasm --"`
-      - Maps the source checkout to `/` in the WASI runtime
-      - Stdlib gets loaded from `/Lib`
-      - Gets `_sysconfigdata__wasi_wasm32-wasi.py` on to `sys.path` via `PYTHONPATH`
-    - Set by `wasi-env`
-      - `WASI_SDK_PATH`
-      - `WASI_SYSROOT`
-      - `CC`
-      - `CPP`
-      - `CXX`
-      - `LDSHARED`
-      - `AR`
-      - `RANLIB`
-      - `CFLAGS`
-      - `LDFLAGS`
-      - `PKG_CONFIG_PATH`
-      - `PKG_CONFIG_LIBDIR`
-      - `PKG_CONFIG_SYSROOT_DIR`
-      - `PATH`
-  - `make all`
-
+That will:
 
-### Running
-
-If you followed the instructions above, you can run the interpreter via e.g., `wasmtime` from within the `Tools/wasi` directory (make sure to set/change `$PYTHON_VERSION` and do note the paths are relative to running in`builddir/wasi` for simplicity only):
+1. Run `configure` for the build Python (same as `wasi.py configure-build-python`)
+2. Run `make` for the build Python (`wasi.py make-build-python`)
+3. Run `configure` for the WASI build (`wasi.py configure-host`)
+4. Run `make` for the WASI build (`wasi.py make-host`)
 
+See the `--help` for the various options available for each of the subcommands which controls things like the location of the WASI SDK, the command to use with the WASI host/runtime, etc. Also note that you can use `--` as a separtor for any of the `configure`-related commands -- including `build` -- to pass arguments to `configure` itself. For example, if you want a pydebug build that also caches the results from `configure`, you can do:
 ```shell
-wasmtime run --mapdir /::../.. --env PYTHONPATH=/builddir/wasi/build/lib.wasi-wasm32-$PYTHON_VERSION python.wasm -- <args>
+python Tools/wasm/wasi.py build -- -C --with-pydebug
 ```
 
-There are also helpers provided by `Tools/wasm/wasm_build.py` as listed below. Also, if you used `Tools/wasm/build_wasi.sh`, a `run_wasi.sh` file will be created in `builddir/wasi` which will run the above command for you (it also uses absolute paths, so it can be executed from anywhere).
-
-#### REPL
-
+The `wasi.py` script is able to infer details from the build Python, and so you only technically need to specify `--with-pydebug` once for `configure-build-python` and `configure-host` will detect its use if you use the discrete steps:
 ```shell
-./Tools/wasm/wasm_build.py wasi repl
+python Tools/wasm/wasi.py configure-build-python -- -C --with-pydebug
+python Tools/wasm/wasi.py make-build-python
+python Tools/wasm/wasi.py configure-host -- -C
+python Tools/wasm/wasi.py make-host
 ```
 
-#### Tests
 
+### Running
+
+If you used `wasi.py` to do your build then there will be a `cross-build/wasm32-wasi/python.sh` file which you can use to run the `python.wasm` file (see the output from the `configure-host` subcommand):
 ```shell
-./Tools/wasm/wasm_build.py wasi test
+cross-build/wasm32-wasi/python.sh --version
 ```
 
-### Debugging
+While you _can_ run `python.wasm` directly, Python will fail to start up without certain things being set (e.g. `PYTHONPATH` for `sysconfig` data). As such, the `python.sh` file records these details for you.
 
-* ``wasmtime run -g`` generates debugging symbols for gdb and lldb. The
-  feature is currently broken, see
-  https://github.com/bytecodealliance/wasmtime/issues/4669 .
-* The environment variable ``RUST_LOG=wasi_common`` enables debug and
-  trace logging.
 
 ## Detect WebAssembly builds
 
@@ -402,15 +368,17 @@ import os, sys
 
 if sys.platform == "emscripten":
     # Python on Emscripten
+    ...
 if sys.platform == "wasi":
     # Python on WASI
+    ...
 
 if os.name == "posix":
     # WASM platforms identify as POSIX-like.
     # Windows does not provide os.uname().
     machine = os.uname().machine
     if machine.startswith("wasm"):
-        # WebAssembly (wasm32, wasm64 in the future)
+        # WebAssembly (wasm32, wasm64 potentially in the future)
 ```
 
 ```python

Tools/wasm/mypy.ini

@@ -1,5 +1,5 @@
 [mypy]
-files = Tools/wasm
+files = Tools/wasm/wasm_*.py
 pretty = True
 show_traceback = True
 
@@ -9,6 +9,3 @@ python_version = 3.8
 # Be strict...
 strict = True
 enable_error_code = truthy-bool,ignore-without-code
-
-# except for incomplete defs, which are useful for module authors:
-disallow_incomplete_defs = False