Skip to content

make regen-all is not deterministic #108740

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
2 tasks done
vstinner opened this issue Aug 31, 2023 · 1 comment
Closed
2 tasks done

make regen-all is not deterministic #108740

vstinner opened this issue Aug 31, 2023 · 1 comment
Labels
type-bug An unexpected behavior, bug, or error

Comments

@vstinner
Copy link
Member

vstinner commented Aug 31, 2023 •
edited by bedevere-bot
Loading

Bug report

Checklist

  • I am confident this is a bug in CPython, not a bug in a third-party project
  • I have searched the CPython issue tracker,
    and am confident this bug has not been reported before

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

A clear and concise description of the bug:

The make regen-all command is not deterministic. Depending in the order in which the target dependencies are executed, the result change.

regen1.sh:

set -e -x

git checkout .
git checkout 79823c103b66030f10e07e04a5462f101674a4fc

# WARNING: it removes all untracked files!
git clean -fdx

./configure --with-pydebug

make regen-cases
make regen-typeslots
make regen-token
make regen-ast
make regen-keyword
make regen-sre
make regen-frozen
make regen-pegen-metaparser
make regen-pegen
make regen-test-frozenmain
make regen-test-levenshtein

make clinic
make regen-global-objects

echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c || true
echo
wc -l Python/deepfreeze/deepfreeze.c

regen2.sh:

set -e -x

git checkout .
git checkout 79823c103b66030f10e07e04a5462f101674a4fc

# WARNING: it removes all untracked files!
git clean -fdx

./configure --with-pydebug

make clinic
make regen-global-objects

make regen-cases
make regen-typeslots
make regen-token
make regen-ast
make regen-keyword
make regen-sre
make regen-frozen
make regen-pegen-metaparser
make regen-pegen
make regen-test-frozenmain
make regen-test-levenshtein

echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c || true
echo
wc -l Python/deepfreeze/deepfreeze.c

regen1.sh output:

(...)

            &_Py_ID(traceback),
            &_Py_ID(traceback),

146433 Python/deepfreeze/deepfreeze.c

regen2.sh output:

(...)

146543 Python/deepfreeze/deepfreeze.c
  • Using regen1.sh, there is a traceback identfier in Python/deepfreeze/deepfreeze.c.
  • Using regen1.sh, there is no traceback identfier in Python/deepfreeze/deepfreeze.c.

These commands are coming from regen-all in Makefile.pre.in:

regen-all: regen-cases regen-typeslots \
        regen-token regen-ast regen-keyword regen-sre regen-frozen clinic \
        regen-pegen-metaparser regen-pegen regen-test-frozenmain \
        regen-test-levenshtein regen-global-objects

Linked PRs
@vstinner vstinner added the type-bug An unexpected behavior, bug, or error label Aug 31, 2023
@vstinner vstinner mentioned this issue Aug 31, 2023
Merged
@vstinner
Copy link
Member Author

Simpler reproducer.

regen1.sh:

set -e -x

make clean

make regen-global-objects
make Python/deepfreeze/deepfreeze.c

echo "=== GLOBAL OBJECTS ==="
echo
grep 'ID(traceback)' ./Include/internal/pycore_global*.h ||:
echo
echo "=== DEEPFREEZE ==="
echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c ||:
echo
wc -l Python/deepfreeze/deepfreeze.c

VS

regen2.sh:

set -e -x

make clean

make Python/deepfreeze/deepfreeze.c
make regen-global-objects

echo "=== GLOBAL OBJECTS ==="
echo
grep 'ID(traceback)' ./Include/internal/pycore_global*.h ||:
echo
echo "=== DEEPFREEZE ==="
echo
grep 'ID(traceback)' Python/deepfreeze/deepfreeze.c ||:
echo
wc -l Python/deepfreeze/deepfreeze.c

Depending if regen-global-objects is run before or after Python/deepfreeze/deepfreeze.c is built / updated, the result is different. I suppose that it's a missing dependency in Makefile.

vstinner added a commit to vstinner/cpython that referenced this issue Aug 31, 2023
@vstinner
pythongh-108740: Fix "make regen-all" race condition
0419d99 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script
  anymore.
* "make regen-deepfreeze" now only updates deepfreeze.c, it doesn't
  build deepfreeze.o anymore.
@bedevere-bot bedevere-bot mentioned this issue Aug 31, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 2, 2023
@vstinner
pythongh-108740: Fix "make regen-all" race condition
5bb8646 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script
  anymore.
* "make regen-deepfreeze" now only updates deepfreeze.c, it doesn't
  build deepfreeze.o anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are outdated, these commands are now safe to
  use.
* Document generates files in Doc/using/configure.rst
vstinner added a commit to vstinner/cpython that referenced this issue Sep 6, 2023
@vstinner
pythongh-108740: Fix "make regen-all" race condition
3afb66f 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script
  anymore.
* "make regen-deepfreeze" now only updates deepfreeze.c, it doesn't
  build deepfreeze.o anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are outdated, these commands are now safe to
  use.
* Document generates files in Doc/using/configure.rst
vstinner added a commit that referenced this issue Sep 6, 2023
@vstinner @erlend-aasland
gh-108740: Fix "make regen-all" race condition (#108741)
db1ee6a 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.
* Document generates files in Doc/using/configure.rst.

Co-authored-by: Erlend E. Aasland <[email protected]>
vstinner added a commit to vstinner/cpython that referenced this issue Sep 6, 2023
@vstinner @erlend-aasland
pythongh-108740: Fix "make regen-all" race condition (python#108741)
d24079c 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.
* Document generates files in Doc/using/configure.rst.

Co-authored-by: Erlend E. Aasland <[email protected]>
(cherry picked from commit db1ee6a)
@bedevere-bot bedevere-bot mentioned this issue Sep 6, 2023
Merged
vstinner added a commit to vstinner/cpython that referenced this issue Sep 6, 2023
@vstinner @erlend-aasland
pythongh-108740: Fix "make regen-all" race condition (python#108741)
f6e56f8 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.

Backport notes:

* Omit Doc/using/configure.rst changes.

Co-authored-by: Erlend E. Aasland <[email protected]>
(cherry picked from commit db1ee6a)
vstinner added a commit to vstinner/cpython that referenced this issue Sep 6, 2023
@vstinner @erlend-aasland
pythongh-108740: Fix "make regen-all" race condition (python#108741)
d1f2ef5 
Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.

Backport notes:

* Omit Doc/using/configure.rst changes.
* no need to change "make clinic", it didn't run
  generate_global_objects.py script before.

Co-authored-by: Erlend E. Aasland <[email protected]>
(cherry picked from commit db1ee6a)
@bedevere-bot bedevere-bot mentioned this issue Sep 6, 2023
Merged
vstinner added a commit that referenced this issue Sep 6, 2023
@vstinner @erlend-aasland
[3.11] gh-108740: Fix "make regen-all" race condition (#108741) (#109021
ed233dd 
)

gh-108740: Fix "make regen-all" race condition (#108741)

Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.

Backport notes:

* Omit Doc/using/configure.rst changes.
* no need to change "make clinic", it didn't run
  generate_global_objects.py script before.

Co-authored-by: Erlend E. Aasland <[email protected]>
(cherry picked from commit db1ee6a)
@vstinner vstinner closed this as completed Sep 6, 2023
Yhg1s pushed a commit that referenced this issue Sep 8, 2023
@vstinner @erlend-aasland
[3.12] gh-108740: Fix "make regen-all" race condition (#108741) (#109019
de0202e 
)

gh-108740: Fix "make regen-all" race condition (#108741)

Fix a race condition in "make regen-all". The deepfreeze.c source and
files generated by Argument Clinic are now generated or updated
before generating "global objects". Previously, some identifiers may
miss depending on the order in which these files were generated.

* "make regen-global-objects": Make sure that deepfreeze.c is
  generated and up to date, and always run "make clinic".
* "make clinic" no longer runs generate_global_objects.py script.
* "make regen-deepfreeze" now only updates deepfreeze.c (C file).
  It doesn't build deepfreeze.o (object) anymore.
* Remove misleading messages in "make regen-global-objects" and
  "make clinic". They are now outdated, these commands are now
  safe to use.
* Document generates files in Doc/using/configure.rst.

Co-authored-by: Erlend E. Aasland <[email protected]>
(cherry picked from commit db1ee6a)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@vstinner