bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers

[vstinner]

These fuctions were undocumented and excluded from the limited C API.

Most names defined by these header files were not prefixed by "Py"
and so could create names conflicts. For example, Python-ast.h
defined a "Yield" macro which was conflict with the "Yield" name used
by the Windows <winbase.h> header.

Use the Python ast module instead.

• Move Include/asdl.h to Include/internal/pycore_asdl.h.
• Move Include/Python-ast.h to Include/internal/pycore_ast.h.
• Remove ast.h header file.
• pycore_symtable.h no longer includes Python-ast.h.

https://bugs.python.org/issue43244

[vstinner]

I searched for asdl.h in the top PyPI 4000 projects, I found 3 projects using it: 2 can be ignored, the only interesting one is typed_ast. IMO typed_ast should not block this PR.

typed_ast-1.4.2.tar.gz: this project contains copies of C and header files of CPython.

ast3/Include/Python-ast.h
ast3/Include/asdl.h
ast3/Include/ast.h
ast3/Include/bitset.h
ast3/Include/errcode.h
ast3/Include/graminit.h
ast3/Include/grammar.h
ast3/Include/node.h
ast3/Include/parsetok.h
ast3/Include/pgenheaders.h
ast3/Include/token.h
ast3/Parser/acceler.c
ast3/Parser/bitset.c
ast3/Parser/grammar.c
ast3/Parser/grammar1.c
ast3/Parser/node.c
ast3/Parser/parser.c
ast3/Parser/parser.h
ast3/Parser/parsetok.c
ast3/Parser/tokenizer.c
ast3/Parser/tokenizer.h
ast3/Python/Python-ast.c
ast3/Python/asdl.c
ast3/Python/ast.c
ast3/Python/graminit.c

This project can be modified to copy Python 3.10 internal header files and be built with Py_BUILD_CORE_MODULE macro defined.

Pygments-2.8.0.tar.gz: it's used in example files to test the project, these files are not built, Pygments colorizes text files.

./tests/examplefiles/make/Makefile:		Include/asdl.h \
./tests/examplefiles/make/Makefile.output:'Include/asdl.h' Text

odfpy-1.4.1.tar.gz: the tarball contains many virtual environment in an hidden .tox/ subdirectory.

[vstinner]

I searched for ast.h using \bast\.h\b regex and for Python-ast.h in the top PyPI 4000 projects, I found:

• the 3 same projects than when I search for asdl.h (typed_ast, Pygments, odfpy)
• jsonnet which contains its own ast.h file: core/ast.h

In short, in the 4000 projects, typed_ast should be the only project impacted by this PR.

typed_ast copies/paste code directly from CPython, so I'm not surprised that it's impacted by changes in the CPython ASDL and AST implementation :-) typed_ast can continue copying code from CPython, but may need to be built with Py_BUILD_CORE_MODULE macro defined.

[vstinner]

Ah, test_asyncio failed on the Windows x64 CI, known random test failure: https://bugs.python.org/issue43539

[vstinner]

pycore_ast.h now contains definitions of functions defined in Python-ast.c and in ast.c. Is that a good thing, or do you prefer to keep two header files separated? I failed to find a better name of the generated header file (old Python-ast.h).

[shihai1991]

pycore_ast.h now contains definitions of functions defined in Python-ast.c and in ast.c. Is that a good thing, or do you prefer to keep two header files separated? I failed to find a better name of the generated header file (old Python-ast.h).

Hm, Can we merge pycore_ast_state.h in pycore_ast.h? There just only have a ast_state in pycore_ast_state.h.

[vstinner]

Hm, Can we merge pycore_ast_state.h in pycore_ast.h? There just only have a ast_state in pycore_ast_state.h.

Yes, but I would prefer to not do that.

pycore_ast_state.h is included by pycore_interp.h and this header file is included by many C files. I would prefer to avoid including pycore_ast.h in C files since this header file is problematic, for example it requires #undef Yield to avoid conflict with <winbase.h>.

[vstinner]

PR rebased on master to fix a conflict on compile.h.

[vstinner]

Removed functions are undocumented. They are also excluded from the limited C API (and from the stable ABI).

I updated the PR to mention that explicitly.

I cannot find removed functions in Doc/c-api/:

• asdl.h: asdl
• Python-ast.h: PyAST_obj2mod, PyAST_Check, PyAST_mod2obj, _Py_Module, mod_ty
• ast.h: PyAST_Validate, PyAST_FromNode, PyAST_FromNodeObject, _PyAST_ExprAsUnicode, _PyAST_GetDocString

[vstinner]

Oops, I fixed a typo: fuctions => functions.