gh-131556: Fix build-details.json Makefile target

[colesbury]

Use the contents of pybuilddir.txt as a prefix for the build-details.json path, if it exists.

• Issue: Makefile always regenerates build-details.json #131556

[colesbury]

Things I've tested locally:

• make clean: still works without errors
• make: generates build-details.json if it doesn't exist
• touch pybuilddir.txt && make: regenerates build-details.json
• make test (after building): no longer regenerates build-details.json if it's up to date

[corona10]

lgtm, it's very interesting approach that the generated Makefile could be different following to pybuilddir.txt :)

@erlend-aasland Do you have better ideas?

[colesbury]

lgtm, it's very interesting approach that the generated Makefile could be different following to pybuilddir.txt :)

The $(shell ...) commands are evaluated when the Makefile is run (not when Makefile is generated from Makefile.pre.in).

[corona10]

Ah thanks you are right, but still good to me

[eli-schwartz]

You cannot use $(shell and should probably forget it exists :)

It is a GNU-ism and will break on any other Make implementation, such as the default Make on BSD.

If cpython were to move to require GNU Make then there are many changes which could be made to this Makefile that would make it significantly smaller, but alas, compatibility...

[FFY00]

@colesbury, sorry for the delay getting to this.

After exploring our options a bit, I believe we should move the pybuilddir.txt value calculation from python -m sysconfig --generate-posix-vars to Makefile.pre.in.

This issue is only a symptom of the bigger issue of us not being able to write Makefile targets for files under PYBUILDDIR, for which we already adopted similar sub-optimal workarounds in other places. This detail of the build process design is mostly historical, and I don't think there should be any reason why we couldn't define a PYBUILDDIR variable to Makefile.pre.in — we should have access to the required information there.

I am gonna open a PR implementing this and see how the buildbots react 😅

[colesbury]

Thanks for the explanations

[eli-schwartz]

This issue is only symptom of the bigger issue of us not being able to write Makefile targets for files under PYBUILDDIR, for which we already adopted similar sub-optimal workarounds in other places. This detail of the build process design is mostly historical, and I don't think there should be any reason why we couldn't add PYBUILDDIR variable to Makefile.pre.in — we should have access to the required information there.

Yup. As long as the information is known and frozen at the time of running ./configure, there's no real reason you can't set Makefile variables at that time instead of using target dynamism.

Makefiles have quite a bit of flexibility for the commands you run during the build, but I unfortunately very little (portable) flexibility in terms of defining the actual graph of the build jobs e.g. target filenames. That is, in very large part, why configure scripts were invented.

I suspect there may not be any use case for pybuilddir.txt to change without a reconfigure. It can be useful for visibility by commands being run (such as the python exe) without imparting meaning to Make itself. Using a Makefile variable $(PYBUILDDIR) seems reasonable to me.

[FFY00]

Yup. As long as the information is known and frozen at the time of running ./configure, there's no real reason you can't set Makefile variables at that time instead of using target dynamism.

Yeah, it is. It's just more annoying to manipulate data there IMO, and depending on exactly what we are doing with the data, it may introduce the need of duplicating it in the Windows build-system. Though, in this case, that isn't a big worry, and even if it was more of a concern, the benefits would still be far worth it.

Makefiles have quite a bit of flexibility for the commands you run during the build, but I unfortunately very little (portable) flexibility in terms of defining the actual graph of the build jobs e.g. target filenames. That is, in very large part, why configure scripts were invented.

There is a tiny bit of wiggle room by using the include directive on files generated by a target, in which case make will generate/remake them before processing the include (explained in https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html). However, that introduces extra complexity, and still isn't very ergonomic, so I like to avoid it when we have another alternative.

I suspect there may not be any use case for pybuilddir.txt to change without a reconfigure. It can be useful for visibility by commands being run (such as the python exe) without imparting meaning to Make itself. Using a Makefile variable $(PYBUILDDIR) seems reasonable to me.

No, indeed, pybuilddir.txt should be constant across the full lifetime of the Makefile.