Support for missing setup.py on startup

[stellaraccident]

Thanks for a great tool - saved me a bunch of time on a couple of projects.

I'm running in to an issue where the result of our build system is a package directory containing a setup.py, which creates a chicken-and-egg problem with cibuildwheel.

I'm doing this:

          package_dir="./iree-install/python_packages/iree_rt"
          export CIBW_BEFORE_BUILD="rm -f ./iree-build/CMakeCache.txt && \
            rm -Rf ./iree-install && \
            python ./main_checkout/build_tools/cmake/cmake_configure_ci.py \
            -B./iree-build -DCMAKE_INSTALL_PREFIX=./iree-install \
            -DCMAKE_BUILD_TYPE=Release \
            -DIREE_BUILD_COMPILER=OFF \
            -DIREE_BUILD_PYTHON_BINDINGS=ON \
            -DIREE_BUILD_SAMPLES=OFF && \
            cmake --build ./iree-build --target install/strip"
          # TODO: cibuildwheel sanity checks this, but our setup.py is the
          # *output* of the build :( File a bug.
          mkdir -p $package_dir && touch $package_dir/setup.py
          python -m cibuildwheel --output-dir wheelhouse $package_dir

I peeked in the source and it looked like the exist check at the top for a valid package file was just a sanity check, so I touched it before invoking to bypass. The result of the CIBW_BEFORE_BUILD will produce it. I verified this works, and due to a bug, I was also not generating valid packages on the first try, and I verified then that the error message downstream when this happens is reasonable (it suggests that a setup file is missing).

Can we just remove the check or at least downgrade it to an informational message?

Also, would be lovely at some point if I could pass multiple package directories and have them built in sequence. Our build system can produce multiple packages and I'm currently having to duplicate build things in a matrix and package them one by one.

[henryiii]

I think touching setup.py is a fine workaround, personally A generated setup.py is already a very sticky situation - you should be driving the compilation from setup.py, not making a setup.py, to be a "nice" Python package. A "nice" Python package is one that will install correctly from SDist if a wheel is missing, using pyproject.toml to ensure dependencies are satisfied. I do understand there are cases where things are not "nice" and would require a lot of work to make it "nice", but I think you should expect a bit of a workaround in that case.

Now if I were you (where "you" is assumed to be both the package author and the person setting up the wheels, which is not always the case), I'd add a setup.cfg and pull out all the not-required-to-be-generated parts and put them in a declarative form there. You have a simpler generation step and the check now passes, as any of the three files need to be present.

Can we just remove the check or at least downgrade it to an informational message?

Nobody reads informational messages. And logs are way too long as it is in many cases. I think an explicit workaround for very unusual situations is better.

Can't you just run cibuildwheel multiple times, do you really need a matrix?

[stellaraccident]

Ok, seems reasonable. I did go back and forth on the sdist bit multiple times and will likely revisit in the future. Suffice to say that this is a project that builds python packages as an ancillary detail and would require some fairly significant inversion to do it the "right way" from the perspective of python packaging.

Regarding invoking multiple times, this mainly comes from the Linux-runs-in-a-container part: it is hard to cache artifacts across invocations the same as you would with other platforms. And this again comes from the above exceptional circumstance, so I get the direction.

[Czaki]

Maybe rethink a bit and create setup.py which will call cmake?

[henryiii]

Keep in mind that with a "proper" python package, one that has a pyproject.toml and sets build-backend = "setuptools.build_meta", the SDist step is always run. You can use this to perform a "collection" of the surrounding package, I would think (though I've never tried, but I've done some tricks similar to this in pybind11). Generally, I place the setup.* and pyproject toml in the main directory, and then put everything else in /python, for a package that is not primarily a Python package. That way it still behaves "nicely" in the Python ecosystem.

See https://github.com/GooFit/GooFit, for example.

[stellaraccident]

Maybe rethink a bit and create setup.py which will call cmake?

Yeah, I had that in an earlier draft. It became unwieldy and I moved on from it, but I may go back to it at some point now that I understand more. Because I have 5 different packages that derive from this same C++ codebase, I have found all ways to do this pretty awkward, involving a lot of repetition one way or the other (or bending things that I don't feel like should need to be bent for the sake of satisfying one language binding).

I have to say that I started very sold on the sdist approach at the outset for the usual reasons (transparency, accessibility, etc), but my project hits up against realities pretty quickly: python packaging is still pretty nascent compared to what I'm used to with more full featured package managers. Particularly, the 1:1 correspondence between setup.py:source_tree:sdist:wheel is fairly limiting. If this were a debian package, I would expect that a complicated project would have a src deb that mapped to the original project source export and produced several binary debs. Further, I would break the project up more and isolate the dependencies into their own dependent packages (vs bundling as submodules) and may have a dev vs regular package, etc. Since I don't get any of that with Python packaging, some relatively impure decisions need to be made somewhere, and I've been struggling to find the right balance.

And we do actually need those package delineations, as they represent legitimate optionality of components, the largest of which is 40x larger than the smallest (minimal needed to run), takes 3+ hours to build, has limited platform support, and requires a choreography of two different build systems (some of our upstream dependencies are quite ill-behaved, and while we try to keep our core system well factored, we have to meet the others where they are at and just try to keep them quarantined).

I don't make any claims about what I've managed to put together except that it works and provides some reasonable boundaries for our project -- which is better than what it was. I have felt along the way that python packaging really exists at a different level of scale to what we need, but I may be missing things (just coming in from the outside, it can be hard to parse the whole history and direction of these things)... Our releases are here. If anyone has ideas, I'm certainly open to hearing them.

[Czaki]

And we do actually need those package delineations, as they represent legitimate optionality of components, the largest of which is 40x larger than the smallest (minimal needed to run), takes 3+ hours to build, has limited platform support, and requires a choreography of two different build systems (some of our upstream dependencies are quite ill-behaved, and while we try to keep our core system well factored, we have to meet the others where they are at and just try to keep them quarantined).

Maybe it is good moment to split package on few smallers parts?

[stellaraccident]

Preaching to the choir :) for everything that we control, we either have split into meaningful pieces or are on a track to do so (working on modularizing llvm now, and it will take some time but is doable). However, the biggest offenders are not within our control, and are annoyingly monolithic and heavily used/needed. For us, the biggest issue is tensorflow, and we keep it carefully cordoned off -- but without drastic surgery, it is essentially unusable with a binary size less than 200MiB. They know it's a problem and any plans to realistically fix it are years off, I would guessb (if ever). So we just deal...

…

On Thu, Jan 7, 2021, 12:55 AM Grzegorz Bokota ***@***.***> wrote: And we do actually need those package delineations, as they represent legitimate optionality of components, the largest of which is 40x larger than the smallest (minimal needed to run), takes 3+ hours to build, has limited platform support, and requires a choreography of two different build systems (some of our upstream dependencies are quite ill-behaved, and while we try to keep our core system well factored, we have to meet the others where they are at and just try to keep them quarantined). Maybe it is good moment to split package on few smallers parts? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#515 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYVAE4T4BBSCVOX735EF3SYVZH7ANCNFSM4VULBJNA> .

[henryiii]

Look at the trick pybind11 uses to split itself into multiple packages from a single source. Modern packages do not need to be 1:1. The setup.py in your source does not need to be the setup.py in your SDist.

[stellaraccident]

It is more subtle than I can fully understand just by looking at it, and will need to actually do a build of the pybind11 packages to understand it (but I think I get the gist). But if I do understand what it is doing, I think my critique of the limitations of python source packaging holds. Do you have any examples of the trick applied to larger projects?

…

On Thu, Jan 7, 2021, 2:26 AM Henry Schreiner ***@***.***> wrote: Look at the trick pybind11 uses to split itself into multiple packages from a single source. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#515 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYVAGAFATE5MFH2COUWULSYWD6BANCNFSM4VULBJNA> .

[henryiii]

The core of the trick (several things are going on there) is that setup.py a) reads /pkg1/setup.py and then uses that in the SDist instead of itself, then b) executes the new setup.py. That way, you can generated different packages from one package. If you have the master setup.py / original git source, you need to use an environment variable to pick which one to build. Once you have the SDists, those don't need special environment variables, they are different packages with different names at that point.

[stellaraccident]

Thanks. I may do something like this for our "rt" package (runtime), which is the core of what is "ours" and would be good to have available as an sdist (it should be able to build anywhere with just source and a compiler). I can also probably get this inversion to make sense for our "compiler" package but have some work to do on the llvm dep. To a first approximation, though, the compiler tools packages are bespoke and have about a 0% chance of building on someone's system that hasn't said the precisely correct incantations. The issues that tensorflow has with this are well documented by the community, and I can say concretely, that the multiple days I spent just getting it to build properly in a manylinux2014 container (versus just calling itself that) is unlikely to work for arbitrary end users without specific interventions.

…

On Thu, Jan 7, 2021, 6:27 AM Henry Schreiner ***@***.***> wrote: The core of the trick (several things are going on there) is that setup.py a) reads /pkg1/setup.py and then uses that in the SDist instead of itself, then b) executes the new setup.py. That way, you can generated different packages from one package. If you have the master setup.py / original git source, you need to use an environment variable to pick which one to build. Once you have the SDists, those don't need special environment variables, they are different packages with different names at that point. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#515 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADYVAC7EPFUQ4IJ2CJKGGDSYXAGLANCNFSM4VULBJNA> .

[henryiii]

I do understand. I still recommend designing to match standard packing expectations as much as possible, and then using cibuildwheel + magic setup incantations / docker images for what you have to do use it for. If you build wheels, that really does cover 95% (or less for manylinux2010 and especially manylinux2014, almost entirely due to old pip versions) of the use cases. (And if you can somehow trigger a nice "Maybe try updating your pip version" in the setup.py on failure with the SDist, you might be able to get a nice user experience for old pip users who can upgrade.)

[joerick]

Thanks for the report @stellaraccident, I think that touch setup.cfg or similar is probably a good workaround for this case.

Closing for now, but good luck and feel free to reopen if you have any more issues :)