Skip to content

Commit 51e8a9a

Browse files
author
Release Manager
committed
gh-40597: Migrate pdf-doc build to meson <!-- ^ Please provide a concise and informative title. --> <!-- ^ Don't put issue numbers in the title, do this in the PR description below. --> <!-- ^ For example, instead of "Fixes #12345" use "Introduce new method to calculate 1 + 2". --> <!-- v Describe your changes below in detail. --> <!-- v Why is this change required? What problem does it solve? --> <!-- v If this PR resolves an open issue, please link to it here. For example, "Fixes #12345". --> Migrate the pdf-doc building to meson. For this, a bug in the builder is fixed that prevented running Make against the sphinx-generated files when invoked via meson. Along the way, the output in case of an error of the make script is improved. Generated pdfs can be found at https://github.com/sagemath/sage/actions/ runs/17005745386/artifacts/3779209846. ### 📝 Checklist <!-- Put an `x` in all the boxes that apply. --> - [ ] The title is concise and informative. - [ ] The description explains in detail what this PR is about. - [ ] I have linked a relevant issue or discussion. - [ ] I have created tests covering the changes. - [ ] I have updated the documentation and checked the documentation preview. ### ⌛ Dependencies <!-- List all open PRs that this PR logically depends on. For example, --> <!-- - #12345: short description why this is a dependency --> <!-- - #34567: ... --> URL: #40597 Reported by: Tobias Diez Reviewer(s): Dima Pasechnik, Tobias Diez, user202729
2 parents 2272ab7 + 4487fc2 commit 51e8a9a

File tree

5 files changed

+91
-113
lines changed

5 files changed

+91
-113
lines changed

‎.github/workflows/doc-build-pdf.yml‎

Lines changed: 51 additions & 106 deletions
Original file line numberDiff line numberDiff line change
@@ -2,148 +2,93 @@ name: Build documentation (PDF)
22

33
on:
44
pull_request:
5+
merge_group:
56
push:
7+
tags:
8+
# Match all release tags including beta, rc
9+
- '[0-9]+.[0-9]+'
10+
- '[0-9]+.[0-9]+.[0-9]+'
11+
- '[0-9]+.[0-9]+.beta[0-9]+'
12+
- '[0-9]+.[0-9]+.[0-9]+.beta[0-9]+'
13+
- '[0-9]+.[0-9]+.rc[0-9]+'
14+
- '[0-9]+.[0-9]+.[0-9]+.rc[0-9]+'
15+
branches:
16+
- develop
617
workflow_dispatch:
718
# Allow to run manually
8-
inputs:
9-
platform:
10-
description: 'Platform'
11-
required: true
12-
default: 'ubuntu-noble-standard'
13-
docker_tag:
14-
description: 'Docker tag'
15-
required: true
16-
default: 'dev'
1719

1820
concurrency:
1921
# Cancel previous runs of this workflow for the same branch
2022
group: ${{ github.workflow }}-${{ github.ref }}
2123
cancel-in-progress: true
2224

2325
env:
24-
# Same as in build.yml
25-
TOX_ENV: "docker-${{ github.event.inputs.platform || 'ubuntu-noble-standard' }}-incremental"
26-
BUILD_IMAGE: "localhost:5000/${{ github.repository }}/sage-${{ github.event.inputs.platform || 'ubuntu-noble-standard' }}-with-targets:ci"
27-
FROM_DOCKER_REPOSITORY: "ghcr.io/sagemath/sage/"
28-
FROM_DOCKER_TARGET: "with-targets"
29-
FROM_DOCKER_TAG: ${{ github.event.inputs.docker_tag || 'dev'}}
30-
EXTRA_CONFIGURE_ARGS: --enable-fat-binary
26+
PYTHON_VERSION: 3.11
3127

3228
jobs:
33-
build-doc-pdf:
29+
doc-pdf:
3430
runs-on: ubuntu-latest
35-
services:
36-
# https://docs.docker.com/build/ci/github-actions/local-registry/
37-
registry:
38-
image: registry:2
39-
ports:
40-
- 5000:5000
4131
steps:
42-
- name: Maximize build disk space
43-
uses: easimon/maximize-build-space@v10
44-
with:
45-
# need space in /var for Docker images
46-
root-reserve-mb: 30000
47-
remove-dotnet: true
48-
remove-android: true
49-
remove-haskell: true
50-
remove-codeql: true
51-
remove-docker-images: true
5232
- name: Checkout
5333
uses: actions/checkout@v4
54-
- name: Install test prerequisites
55-
# From docker.yml
56-
run: |
57-
sudo DEBIAN_FRONTEND=noninteractive apt-get update
58-
sudo DEBIAN_FRONTEND=noninteractive apt-get install tox
59-
sudo apt-get clean
60-
df -h
34+
6135
- name: Merge CI fixes from sagemath/sage
6236
run: |
63-
mkdir -p upstream
64-
.github/workflows/merge-fixes.sh 2>&1 | tee upstream/ci_fixes.log
37+
.github/workflows/merge-fixes.sh
6538
env:
6639
GH_TOKEN: ${{ github.token }}
67-
SAGE_CI_FIXES_FROM_REPOSITORIES: ${{ vars.SAGE_CI_FIXES_FROM_REPOSITORIES }}
68-
69-
# Building
70-
71-
- name: Generate Dockerfile
72-
# From docker.yml
73-
run: |
74-
tox -e ${{ env.TOX_ENV }}
75-
cp .tox/${{ env.TOX_ENV }}/Dockerfile .
76-
env:
77-
# Only generate the Dockerfile, do not run 'docker build' here
78-
DOCKER_TARGETS: ""
7940

80-
- name: Set up Docker Buildx
81-
uses: docker/setup-buildx-action@v3
41+
- name: Cache conda packages
42+
uses: actions/cache@v4
8243
with:
83-
driver-opts: network=host
84-
85-
- name: Build Docker image
86-
id: image
87-
uses: docker/build-push-action@v6
44+
path: ~/conda_pkgs_dir
45+
key:
46+
${{ runner.os }}-conda-${{ hashFiles('environment-${{ env.PYTHON_VERSION }}-linux.yml') }}
47+
48+
- name: Compiler cache
49+
uses: hendrikmuhs/[email protected]
8850
with:
89-
# push and load may not be set together at the moment
90-
push: true
91-
load: false
92-
context: .
93-
tags: ${{ env.BUILD_IMAGE }}
94-
target: with-targets
95-
cache-from: type=gha
96-
cache-to: type=gha,mode=max
97-
build-args: |
98-
NUMPROC=6
99-
USE_MAKEFLAGS=-k V=0 SAGE_NUM_THREADS=4 --output-sync=recurse
100-
TARGETS_PRE=build/make/Makefile
101-
TARGETS=ci-build-with-fallback
102-
103-
- name: Start container
104-
id: container
105-
# Try to continue when "exporting to GitHub Actions Cache" failed with timeout
106-
if: (success() || failure())
107-
run: |
108-
docker run --name BUILD -dit \
109-
--mount type=bind,src=$(pwd),dst=$(pwd) \
110-
--workdir $(pwd) \
111-
${{ env.BUILD_IMAGE }} /bin/sh
51+
key: ${{ runner.os }}-meson-${{ env.PYTHON_VERSION }}
11252

113-
#
114-
# On PRs and pushes to branches
115-
#
116-
117-
- name: Update system packages
118-
id: packages
119-
if: (success() || failure()) && steps.container.outcome == 'success'
53+
- name: Setup Conda environment
54+
uses: conda-incubator/setup-miniconda@v3
55+
with:
56+
python-version: ${{ env.PYTHON_VERSION }}
57+
# Disabled for now due to
58+
# https://github.com/conda-incubator/setup-miniconda/issues/379
59+
# miniforge-version: latest
60+
use-mamba: true
61+
channels: conda-forge
62+
channel-priority: true
63+
activate-environment: sage-dev
64+
environment-file: environment-${{ env.PYTHON_VERSION }}-linux.yml
65+
66+
- name: Build Sage
67+
shell: bash -l {0}
12068
run: |
121-
export PATH="build/bin:$PATH"
122-
eval $(sage-print-system-package-command auto update)
123-
eval $(sage-print-system-package-command auto --yes --no-install-recommends install zip)
124-
eval $(sage-print-system-package-command auto --spkg --yes --no-install-recommends install git texlive texlive_luatex free_fonts xindy)
125-
shell: sh .github/workflows/docker-exec-script.sh BUILD /sage {0}
69+
export PATH="/usr/lib/ccache:/usr/local/opt/ccache/libexec:$PATH"
70+
export CC="ccache $CC"
71+
export CXX="ccache $CXX"
72+
pip install --no-build-isolation --config-settings=builddir=builddir . -v
12673
127-
- name: Build doc (PDF)
128-
id: docbuild
129-
if: (success() || failure()) && steps.packages.outcome == 'success'
74+
- name: Build documentation
75+
shell: bash -l {0}
13076
run: |
131-
export MAKE="make -j5 --output-sync=recurse" SAGE_NUM_THREADS=5
132-
make doc-clean doc-uninstall; make sagemath_doc_html-build-deps sagemath_doc_pdf-no-deps
133-
shell: sh .github/workflows/docker-exec-script.sh BUILD /sage {0}
77+
sudo DEBIAN_FRONTEND=noninteractive sudo apt-get update
78+
sudo DEBIAN_FRONTEND=noninteractive apt-get install $(build/bin/sage-get-system-packages debian texlive texlive_luatex free_fonts xindy)
79+
meson compile -C builddir doc-pdf
80+
env:
81+
SAGE_DOCBUILD_OPTS: "--include-tests-blocks"
13482

13583
- name: Copy doc
13684
id: copy
137-
if: (success() || failure()) && steps.docbuild.outcome == 'success'
13885
run: |
13986
mkdir -p ./doc
140-
# We copy everything to a local folder
141-
docker cp BUILD:/sage/local/share/doc/sage/pdf doc
87+
cp -r builddir/src/doc/pdf doc/
14288
# Zip everything for increased performance
14389
zip -r doc-pdf.zip doc
14490
14591
- name: Upload doc
146-
if: (success() || failure()) && steps.copy.outcome == 'success'
14792
uses: actions/upload-artifact@v4
14893
with:
14994
name: doc-pdf

‎.github/workflows/doc-build.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ env:
2626
PYTHON_VERSION: 3.11
2727

2828
jobs:
29-
build-doc:
29+
doc-html:
3030
runs-on: ubuntu-latest
3131
steps:
3232
- name: Checkout

‎pyproject.toml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,13 @@ extra = [
205205
"pkg:generic/qepcad",
206206
"pkg:generic/tides",
207207
]
208+
# Uncommented for now as most devs probably don't need to run it locally
209+
# docs-pdf = [
210+
# "pkg:generic/latexmk",
211+
# "pkg:generic/lualatex",
212+
# "pkg:fonts/freefonts",
213+
# "pkg:generic/xindy",
214+
# ]
208215

209216
[dependency-groups]
210217
dev = ["conda-lock", "grayskull", "meson", "pygithub", "toml", "tqdm", "uv"]

‎src/sage_docbuild/builders.py‎

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,7 @@
6666
import os
6767
import pickle
6868
import re
69+
import shlex
6970
import shutil
7071
import subprocess
7172
import sys
@@ -265,13 +266,37 @@ def pdf(self):
265266
with open(tex_file, 'w') as f:
266267
f.write(ref)
267268

268-
make_target = "cd '%s' && $MAKE %s && mv -f *.pdf '%s'"
269-
error_message = "failed to run $MAKE %s in %s"
270-
command = 'all-pdf'
269+
make_cmd = os.environ.get('MAKE', 'make')
270+
command = shlex.split(make_cmd) + ['all-pdf']
271+
logger.debug(f"Running {' '.join(command)} in {tex_dir}")
272+
273+
proc = subprocess.run(
274+
command,
275+
check=False, cwd=tex_dir,
276+
capture_output=True,
277+
text=True,
278+
)
279+
280+
if proc.returncode != 0:
281+
logger.error(f"stdout from {make_cmd}:\n{proc.stdout}")
282+
logger.error(f"stderr from {make_cmd}:\n{proc.stderr}")
283+
raise RuntimeError(f"failed to run {' '.join(command)} in {tex_dir}")
284+
285+
if proc.stdout:
286+
logger.debug(f"make stdout:\n{proc.stdout}")
287+
if proc.stderr:
288+
# Still surface stderr even on success, but at debug level
289+
logger.debug(f"make stderr:\n{proc.stderr}")
290+
291+
# Move generated PDFs
292+
for pdf in tex_dir.glob("*.pdf"):
293+
try:
294+
shutil.move(str(pdf), pdf_dir)
295+
except Exception as e:
296+
logger.error(f"Failed moving {pdf} to {pdf_dir}: {e}")
297+
raise
271298

272-
if subprocess.call(make_target % (tex_dir, command, pdf_dir), close_fds=False, shell=True):
273-
raise RuntimeError(error_message % (command, tex_dir))
274-
logger.warning(f"Build finished. The built documents can be found in {pdf_dir}.")
299+
logger.info(f"Build finished. The built documents can be found in {pdf_dir}.")
275300

276301
def clean(self, *args):
277302
shutil.rmtree(self._doctrees_dir())

‎tools/update-conda.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,7 @@ def get_dependencies(pyproject_toml: Path, python: str, platform: str) -> set[st
168168
.replace("symengine", "python-symengine")
169169
.replace("memory_allocator", "memory-allocator")
170170
.replace("pkg:generic/r-lattice", "r-lattice")
171+
.replace("pkg:generic/latexmk", "latexmk")
171172
for req in all_requirements
172173
}
173174
# Exclude requirements not available on conda (for a given platform)

0 commit comments

Comments
 (0)