Skip to content

Commit db1ee6a

Browse files
gh-108740: Fix "make regen-all" race condition (#108741)
Fix a race condition in "make regen-all". The deepfreeze.c source and files generated by Argument Clinic are now generated or updated before generating "global objects". Previously, some identifiers may miss depending on the order in which these files were generated. * "make regen-global-objects": Make sure that deepfreeze.c is generated and up to date, and always run "make clinic". * "make clinic" no longer runs generate_global_objects.py script. * "make regen-deepfreeze" now only updates deepfreeze.c (C file). It doesn't build deepfreeze.o (object) anymore. * Remove misleading messages in "make regen-global-objects" and "make clinic". They are now outdated, these commands are now safe to use. * Document generates files in Doc/using/configure.rst. Co-authored-by: Erlend E. Aasland <[email protected]>
1 parent a0773b8 commit db1ee6a

File tree

4 files changed

+38
-10
lines changed

4 files changed

+38
-10
lines changed

Doc/using/configure.rst

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,29 @@ See also :pep:`7` "Style Guide for C Code" and :pep:`11` "CPython platform
6060
support".
6161

6262

63+
Generated files
64+
===============
65+
66+
To reduce build dependencies, Python source code contains multiple generated
67+
files. Commands to regenerate all generated files::
68+
69+
make regen-all
70+
make regen-stdlib-module-names
71+
make regen-limited-abi
72+
make regen-configure
73+
74+
The ``Makefile.pre.in`` file documents generated files, their inputs, and tools used
75+
to regenerate them. Search for ``regen-*`` make targets.
76+
77+
The ``make regen-configure`` command runs `tiran/cpython_autoconf
78+
<https://github.com/tiran/cpython_autoconf>`_ container for reproducible build;
79+
see container ``entry.sh`` script. The container is optional, the following
80+
command can be run locally, the generated files depend on autoconf and aclocal
81+
versions::
82+
83+
autoreconf -ivf -Werror
84+
85+
6386
.. _configure-options:
6487

6588
Configure Options

Makefile.pre.in

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -490,6 +490,7 @@ OBJECT_OBJS= \
490490
Objects/weakrefobject.o \
491491
@PERF_TRAMPOLINE_OBJ@
492492

493+
DEEPFREEZE_C = Python/deepfreeze/deepfreeze.c
493494
DEEPFREEZE_OBJS = Python/deepfreeze/deepfreeze.o
494495

495496
##########################################################################
@@ -777,7 +778,6 @@ coverage-report: regen-token regen-frozen
777778
.PHONY: clinic
778779
clinic: check-clean-src $(srcdir)/Modules/_blake2/blake2s_impl.c
779780
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/clinic/clinic.py --make --exclude Lib/test/clinic.test.c --srcdir $(srcdir)
780-
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py
781781

782782
.PHONY: clinic-tests
783783
clinic-tests: check-clean-src $(srcdir)/Lib/test/clinic.test.c
@@ -1252,12 +1252,12 @@ regen-frozen: Tools/build/freeze_modules.py $(FROZEN_FILES_IN)
12521252
# Deepfreeze targets
12531253

12541254
.PHONY: regen-deepfreeze
1255-
regen-deepfreeze: $(DEEPFREEZE_OBJS)
1255+
regen-deepfreeze: $(DEEPFREEZE_C)
12561256

12571257
DEEPFREEZE_DEPS=$(srcdir)/Tools/build/deepfreeze.py Include/internal/pycore_global_strings.h $(FREEZE_MODULE_DEPS) $(FROZEN_FILES_OUT)
12581258

12591259
# BEGIN: deepfreeze modules
1260-
Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)
1260+
$(DEEPFREEZE_C): $(DEEPFREEZE_DEPS)
12611261
$(PYTHON_FOR_FREEZE) $(srcdir)/Tools/build/deepfreeze.py \
12621262
Python/frozen_modules/importlib._bootstrap.h:importlib._bootstrap \
12631263
Python/frozen_modules/importlib._bootstrap_external.h:importlib._bootstrap_external \
@@ -1284,8 +1284,6 @@ Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)
12841284
Python/frozen_modules/frozen_only.h:frozen_only \
12851285
-o Python/deepfreeze/deepfreeze.c
12861286
# END: deepfreeze modules
1287-
@echo "Note: Deepfreeze may have added some global objects,"
1288-
@echo " so run 'make regen-global-objects' if necessary."
12891287

12901288
# We keep this renamed target around for folks with muscle memory.
12911289
.PHONY: regen-importlib
@@ -1294,11 +1292,12 @@ regen-importlib: regen-frozen
12941292
############################################################################
12951293
# Global objects
12961294

1295+
# Dependencies which can add and/or remove _Py_ID() identifiers:
1296+
# - deepfreeze.c
1297+
# - "make clinic"
12971298
.PHONY: regen-global-objects
1298-
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py
1299+
regen-global-objects: $(srcdir)/Tools/build/generate_global_objects.py $(DEEPFREEZE_C) clinic
12991300
$(PYTHON_FOR_REGEN) $(srcdir)/Tools/build/generate_global_objects.py
1300-
@echo "Note: Global objects can be added or removed by other tools (e.g. deepfreeze), "
1301-
@echo " so be sure to re-run regen-global-objects after those tools."
13021301

13031302
############################################################################
13041303
# ABI
@@ -1320,9 +1319,10 @@ regen-limited-abi: all
13201319
############################################################################
13211320
# Regenerate all generated files
13221321

1322+
# "clinic" is regenerated implicitly via "regen-global-objects".
13231323
.PHONY: regen-all
13241324
regen-all: regen-cases regen-typeslots \
1325-
regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \
1325+
regen-token regen-ast regen-keyword regen-sre regen-frozen \
13261326
regen-pegen-metaparser regen-pegen regen-test-frozenmain \
13271327
regen-test-levenshtein regen-global-objects
13281328
@echo
@@ -2597,6 +2597,7 @@ recheck:
25972597
autoconf:
25982598
(cd $(srcdir); autoreconf -ivf -Werror)
25992599

2600+
# See https://github.com/tiran/cpython_autoconf container
26002601
.PHONY: regen-configure
26012602
regen-configure:
26022603
@if command -v podman >/dev/null; then RUNTIME="podman"; else RUNTIME="docker"; fi; \
Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
Fix a race condition in ``make regen-all``. The ``deepfreeze.c`` source and
2+
files generated by Argument Clinic are now generated or updated before
3+
generating "global objects". Previously, some identifiers may miss depending
4+
on the order in which these files were generated. Patch by Victor Stinner.

Tools/build/freeze_modules.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -585,7 +585,7 @@ def regen_makefile(modules):
585585
pyfiles = []
586586
frozenfiles = []
587587
rules = ['']
588-
deepfreezerules = ["Python/deepfreeze/deepfreeze.c: $(DEEPFREEZE_DEPS)",
588+
deepfreezerules = ["$(DEEPFREEZE_C): $(DEEPFREEZE_DEPS)",
589589
"\t$(PYTHON_FOR_FREEZE) $(srcdir)/Tools/build/deepfreeze.py \\"]
590590
for src in _iter_sources(modules):
591591
frozen_header = relpath_for_posix_display(src.frozenfile, ROOT_DIR)

0 commit comments

Comments
 (0)