Future of libcrypt.so.1

[mitsuhiko]

While libcrypt.so.1 really should be part of a linux installation, it seems like plenty of users don't have it. I wonder if it makes sense to consider dropping that dependency or working around it.

It's missing in archlinux, and apparently some WSL2 users also report not having it: astral-sh/rye#258

[indygreg]

#113 has the prior discussion of this. tl;dr is the Linux Standard Base Core Specification purports to require this library. But the language it uses is shall and apparently nobody really cares about LSB compliance anyway.

Since Linux distributions are obviously removing this library, it forces our hand to either remove it completely or separate out the part requiring it (the _crypt) extension module to be a standalone extension module library and not statically linked into libpython.

FWIW I've been slowly coming around to the idea that aggressive static linking of extension modules is a mistake, at least for some build configuration. The use cases that python-build-standalone is growing into warrant stronger compatibility with the way CPython is typically built. And that means building extension modules as shared libraries and possibly shipping standalone shared libraries for shared libraries (like OpenSSL) as well.

[bluss]

Python 3.13 removes crypt from stdlib, which at least means this issue will go away eventually, I guess? https://discuss.python.org/t/pep-594-has-been-implemented-python-3-13-removes-20-stdlib-modules/

[indygreg]

Yeah, I guess it does go away naturally. Unfortunately we have 2-3 years before 3.13 gains meaningful adoption. We need to ship a change prior to 3.13 or else PBS won't be usable for many end-users.

[bfredl]

A compatible libcrypt.so.1 should be included as python/lib/libcrypt.so.1 (or perhaps link it into libpython) . If you rely on shared libraries from the distro for third-party libs like this, your distribution is not in fact standalone (sorry I don't make the rules).

[mitsuhiko]

If you rely on shared libraries from the distro for third-party libs like this, your distribution is not in fact standalone (sorry I don't make the rules).

You need to draw the line somewhere. Historically libcrypt.so.1 was there, it's a rather recent "innovation" that it's not. Unfortunately linux makes it very hard to have a stable user land :(

[bfredl]

You need to draw the line somewhere.

I agree, but typically the line for a "standalone" executable has been the libraries provided by glibc itself, as glibc is needed to bootstrap the use of dynamic modules and shared libraries at all. libcrypt is the library this links to beyond that.

Historically libcrypt.so.1 was there, it's a rather recent "innovation" that it's not.

For better or worse, this is how SONAME:s work on linux. Distros can ship new ABI versions as long as the distro has updated all their own packages.

[indygreg]

Relying on existence of libcrypt.so.1 (and other libraries covered by the Linux Standard Base specifications, such as libc.so.6) has been a solid strategy for >10 years. While Linux user space isn't really stable, the LSB specifications are supposed to define a set of stable libraries and APIs.

What's changed in the past several months is a bunch of distros appear to be chucking glibc's libcrypt.so.1 out the window in favor of libxcrypt and discarding compliance with the LSB specifications in the process. (I know glibc has talked about removing libcrypt but I'm not sure if they are going through with it or the distros all see the writing on the wall and are doing this preemptively.)

I've also heard that distros don't really care about LSB compliance these days anyway and LSB is effectively in a zombie state. I'm not aware of a viable alternative. This is really a shame because it makes it vastly more difficult to construct portable Linux binaries since there are no ~universally agreed upon standards for what is in Linux user space.

typically the line for a "standalone" executable has been the libraries provided by glibc itself

Historically libcrypt.so.1 was provided by glibc. Its presence was as constant as libc.so.6! Hence why I thought it was safe to require.

Anyway, I started hacking on patches to make _crypt a standalone shared library. It isn't trivially straightforward given all the patches we're applying to statically linked extension modules. I'm really tempted to just delete _crypt completely. I wonder if anyone will notice...

[mitsuhiko]

I'm really tempted to just delete _crypt completely. I wonder if anyone will notice...

I believe there are a handful of operations related tools that might benefit of having it, but the question is really if these builds want to go down that path. In particular I know that saltstack and a few other tools of that nature uses the _crypt module.

That said, a lot of tools were already written to catch crypt not being installed and gradually degrade as some linux distributions historically already split off that module into it's own package.

[indygreg]

As the GitHub issue timeline shows, I think I'm close to a working patch to move _crypt out of libpython and into a standalone shared library.

[rmartin16]

Was this dependency removed for all released Python versions?

When trying to use 3.8, 3.9, and 3.10 from releases 20231002 or 20230826, I experience the same issue with libcrypt.so.1. I can certainly provide more details in a new ticket...I just wanted to confirm this change was intended to apply to all versions or not first.

[indygreg]

Yes, the intent was for all Python versions to only link libcrypt.so.1 via the _crypt.so extension module and not any other ELF binary.