Please drop setuptools_scm

[ddevault]

This kind of release process is a huge nuisance for those of us packaging software downstream. Not being able to use PyPI tarballs or even GitHub tarballs is requires us to go well out of our way to accomodate your package. Please just conform to normal Python packaging norms.

[FFY00]

Just set the SETUPTOOLS_SCM_PRETEND_VERSION environment variable.

https://github.com/archlinux/svntogit-community/blob/19f4a531a1cc0b1dedc7aa2a2911d79a610b3caf/trunk/PKGBUILD#L18

[ddevault]

I don't think that's a very good workaround for not having good packaging practices.

[FFY00]

Respectfully, I disagree. I understand that it is annoying but that environment variable is there for just that, to override the setuptools_scm version. We could document this better, but overall I don't really see an issue with this approach besides being slightly annoying to packagers, it has absolutely nothing to do with good or bad packaging practices.

[ddevault]

being slightly annoying to packagers

has absolutely nothing to do with good or bad packaging practices

Is this not a contradiction? There's little reason to rock the boat. Just ship with standard Python packaging tools and it'll work better for everyone.

[FFY00]

How is something being slightly annoying to packagers a bad packaging practice??
First of all, "packaging practices" are executed by packagers, not by upstreams. All you can argue is that the upstream is making the packagers' lives a harder, which can be a fair point, but generally is a tradeoff, so it is up to the upstream to choose how much of their workflow they are willing to sacrifice to make your life easier.

Upstreams are free to manage their projects as they want, but I think they should be careful to not make the workflow incompatible, or unreasonable, with popular packaging workflow requirements. This definitely not the case here, I would agree with you if this did actually make it impossible or unreasonable for you to package the project, but that is just not true.

And please note, I am a packager myself. There's little point trying to argue in name of packagers(TM) here.

Just ship with standard Python packaging tools and it'll work better for everyone.

Just wait until you hear about PEP 517....

[ddevault]

Just wait until you hear about PEP 517....

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

And please note, I am a packager myself.

Debian agrees with me:

https://wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

[eli-schwartz]

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

On that note, in search of a solution I wrote to the git mailing list fairly recently: https://public-inbox.org/git/4f65f02c-1d16-aa2c-3e7b-28d807b9ebe9@archlinux.org/T/

It's highly annoying that github archives are rendered either entirely useless or obscure and difficult to work with, depending on which particular variety of broken setuptools plugin any given project is using. I'll admit that one could do much worse than setuptools_scm, but that still isn't especially ideal IMO.

And PyPI sdist are often not an option as they more often than not don't distribute tests... (This is not a comment on this specific project.)

[FFY00]

Oh, we all know Python is a nightmare. We should all do our part by doing our best to deal with it by making things as uncomplicated and predictable as possible.

We do, but things are very complicated.

Debian agrees with me:

wiki.debian.org/UpstreamGuide#Out-of-VCS_Builds

So if packagers are concerned, my argument has a majority. I've said my piece. I think that you are making a poor packaging choice as an upstream. If you won't change it, there's nothing I can do about it.

That at most only partially agrees with you. The VCS or code repository is not required to be present, it is the one of the resorts when detecting the version.
I understand your annoyance, but it is not really not up to you, or me for that matter, to decide what they are willing to compromise at. As a packager you need to accept when upstream are not willing to make the compromise you are looking for.
If you want to see my position as a lead maintainer, you can check out pypa/build#170. I get your sentiment here, I really do, I just don't think your attitude of "just conform", "this is incompatible" and "it'll work better for everyone" is correct.

[ddevault]

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

pypa/build#170

Funny that you went from -2 to being a part of the problem.

[FFY00]

I would hardly call SETUPTOOLS_SCM_PRETEND_VERSION "ergonomic to use", so it would be nice if we didn't need this sort of awkward workaround.

It sure would, but it is up to the maintainers, not you. As a packager I find this solution annoying, but acceptable, I just fix my PKGBUILD and go around my way.

I'll also add that one small annoyance for you as an upstream maintainer (e.g. having to bump the version in a file somewhere) is N, but package maintainers have to deal with N² annoying packages. It adds up for us, and doesn't add up for you.

I am very aware of the impact of this. It adds up, but it's a one time fix.

Funny that you went from -2 to being a part of the problem.

I am not the main maintainer of this repo, I have only tagged one release, it is not my workflow that this will be breaking.

You can wait for @jaraco to weigh in, but I doubt he'll agree to this, hence my first comment.

[ddevault]

I don't think @jaraco expected to receive pushback for this, and I would hope that the knee-jerk reaction to dismiss these complaints out of hand isn't what we'll get. But @jaraco, please weigh carefully whether or not the perceived benefits of using scmtools-scm, which is almost universally hated among packagers, are worth the trade-off.

[FFY00]

I was never trying to dismiss complaints here, my first comment should have been worded better, sorry. I was trying to provide you an immediate solution, you then escalated saying how this is bad packaging practices.
I am +1 on dropping setuptools_scm, but as I said, it is definitely not my call.

[jaraco]

I appreciate you bringing this concern to bear. Unfortunately, I'll probably need more information to understand what's causing you problems. Since the request here applies to the hundreds of projects I maintain, I'd wish to move it to jaraco/skeleton, where these practices are maintained across those projects, but that's another org, so I can't and I'll just address the concerns here.

Not being able to use PyPI tarballs

You should be able to use PyPI tarballs.

~ $ pip-run --no-binary importlib_resources importlib_resources
Collecting importlib_resources
  Downloading importlib_resources-5.1.0.tar.gz (31 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.1.0-py3-none-any.whl size=24554 sha256=9e6c5526f11d7b45fb761fa597bd2990d973c2dfecfb79bfaffbb7712e654521
  Stored in directory: /Users/jaraco/Library/Caches/pip/wheels/a0/82/c0/6455ddb59a10e136e92e902c63b8864c52c8a8e8146c377766
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.1.0
WARNING: You are using pip version 21.0; however, version 21.0.1 is available.
You should consider upgrading via the '/Users/jaraco/.local/homebrew/opt/python@3.9/bin/python3.9 -m pip install --upgrade pip' command.
Python 3.9.1 (default, Jan  5 2021, 11:10:38) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 

Under what conditions are you unable to install PyPI tarballs?

Please just conform to normal Python packaging norms.

The setuptools_scm approach is a Python packaging norm that's evolved from best practices that go back to core features in setuptools from 2006 and that have been consolidated in the setuptools_scm under the PyPA umbrella as the blessed approach for leveraging SCM metadata in Python project development. I believe other PyPA projects include this behavior, even built-in (though I'm unfamiliar). I acknowledge that it has some shortcomings (the main one being that Github tarballs don't bring enough metadata), but Github checkouts are still supported:

~ $ pip-run git+https://github.com/python/importlib_resources@v5.0.1 -- -c pass
Collecting git+https://github.com/python/importlib_resources@v5.0.1
  Cloning https://github.com/python/importlib_resources (to revision v5.0.1) to /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git clone -q https://github.com/python/importlib_resources /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-req-build-pmqcuum9
  Running command git checkout -q ecc2a24af948c55ecc88f0c3cfc4c3900c8f6246
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: importlib-resources
  Building wheel for importlib-resources (PEP 517) ... done
  Created wheel for importlib-resources: filename=importlib_resources-5.0.1-py3-none-any.whl size=23364 sha256=5fff94d19749d20910aeee76f8ee5e39504e530060591bc3191e1c6b1b5d3b8d
  Stored in directory: /private/var/folders/03/7l0ffypn50b83bp0bt07xcch00n8zm/T/pip-ephem-wheel-cache-2oypcf6_/wheels/0f/33/ae/fa8051a526161d3dfd5c2f824a228aea20468ce132c34a3dfa
Successfully built importlib-resources
Installing collected packages: importlib-resources
Successfully installed importlib-resources-5.0.1

I'm heavily invested in setuptools_scm and the value it provides, and in my experience, the value far outweighs the limitations it imposes. The benefits include:

• No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.
• Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.
• Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).
• Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.
• There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

Removing setuptools_scm support from this project and the hundreds of others I maintain would add toilsome steps, redundant data, noisy commits, inconsistent states, and reduce the fidelity of the intermediate versions produced.

Can you explain more about what workflows are unmet by the current approach? Perhaps there are other approaches that would satisfy the need.

[FFY00]

Under what conditions are you unable to install PyPI tarballs?

The issue is building from github tarballs, as is common for packagers to do. It is usually desired to fetch from the original source rather than a 3rd party service, but I'll let Drew elaborate on this if he wants.

[ddevault]

Lot to unpack here.

You should be able to use PyPI tarballs.

Kind of, though I ran into issues. Crucially, you can't use the PyPI tarballs combined with a standard Python packaging procedure, nor the GitHub tarballs.

Note as well that no one uses pip for packaging Python stuff downstream. Working around the problem in pip definitely doesn't work for us. For one, we have a hard constraint on not accessing the network during the build process.

No need to maintain a manifest.in. Files included in the repo are included in the sdist. No opportunity to add to one and not the other.

Not that hard. I can offer some advice if you find this difficult.

Possible to create releases without a terminal. Simply tagging a commit in the repo, even through the web, creates a release. That means I can accept a pull request, tag it, and cut a release of the changes from my mobile while in bed. Super low friction.

Honestly this sounds irresponsible as hell. This leads to broken releases. Surely you've broken a release doing it this way and had to hastily ship a fix? Strong NACK to this as an advantage. I don't want you releasing things into my production systems from the comfort of your bed.

Repo state and published releases state is kept in sync (it's not possible to have one version declared in the repository and a different version in the tags).

This can also be dealt with. I have a script for doing releases that deals with this stuff.

Intermediate commits between releases indicate their intermediate state instead of falsely declaring a prior or future version. The intermediate state also indicates distance from the last release and the exact git hash of the code at the time it was built.

???

There's no need for any additional development tooling (bump2version, ...), tooling which itself needs to be installed, maintained, etc.

No, just downstream workarounds which also have to be maintained.

I don't really want to engage with you on this discussion further. I'm not going to waste my time when I'm up against the effort of getting you to change your approach for "hundreds of projects". I don't think I could possibly make an argument which would overcome your change resistance when the change involved requires that much work.