Skip to content

FreeBSD support (fix ready) #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LukaszMoskala opened this issue Dec 6, 2023 · 16 comments
Closed

FreeBSD support (fix ready) #17

LukaszMoskala opened this issue Dec 6, 2023 · 16 comments

Comments

@LukaszMoskala
Copy link
Contributor

Hello,

I get NotImplementedError: Unsupported platform: freebsd13 when doing pip install --upgrade homeassistant. Turns out this package recently became dependency of another dependency of homeassistant.

Please replace in setup.py

SYSTEM_IS_UNIX = (sys.platform.startswith("linux") or
                  sys.platform.startswith("darwin"))

with:

SYSTEM_IS_UNIX = (sys.platform.startswith("linux") or
                  sys.platform.startswith("darwin") or
                  sys.platform.startswith("freebsd"))

Probably we also should handle openbsd, netbsd and dragonflybsd, but I didn't verify if these names are correct. Or maybe we should just do:

SYSTEM_IS_UNIX = (sys.platform.startswith("linux") or
                  sys.platform.startswith("darwin") or
                  'bsd' in sys.platform)

I can also confirm that after making my change, package successfully installs on freebsd.

Thank you in advance, let me know if you need anything else. I can open pull request if that's easier for you.

@LukaszMoskala LukaszMoskala mentioned this issue Dec 7, 2023
2 tasks
@rhpvorderman
Copy link
Contributor

Hi. Sure, if that works I can implement that. Thanks for the suggestion!

@ryanwinter
Copy link

I'm still having issues when installing wheel for this package:

 OverflowError: could not allocate a large enough CPU set

@ryanwinter
Copy link

Processing /opt/python-zlib-ng
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: zlib-ng
  Building wheel for zlib-ng (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for zlib-ng (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [74 lines of output]
      /tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/dist.py:314: InformationOnly: Normalizing '0.3.0-dev' to '0.3.0.dev0'
        self.metadata.version = self._normalize_version(self.metadata.version)
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311
      creating build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng
      copying src/zlib_ng/__init__.py -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng
      copying src/zlib_ng/gzip_ng.py -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng
      copying src/zlib_ng/zlib_ng.pyi -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng
      copying src/zlib_ng/py.typed -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng
      creating build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng/zlib-ng
      copying src/zlib_ng/zlib-ng/LICENSE.md -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng/zlib-ng
      copying src/zlib_ng/zlib-ng/README.md -> build/lib.freebsd-13.1-RELEASE-p7-amd64-cpython-311/zlib_ng/zlib-ng
      running build_ext
      Traceback (most recent call last):
        File "/opt/homeassistant.2023.12/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/opt/homeassistant.2023.12/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/opt/homeassistant.2023.12/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 389, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 123, in <module>
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 88, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
          self._build_extensions_serial()
        File "/tmp/pip-build-env-o8jrof8i/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
          self.build_extension(ext)
        File "<string>", line 67, in build_extension
        File "<string>", line 99, in build_zlib_ng
      OverflowError: could not allocate a large enough CPU set
      [end of output]

@rhpvorderman
Copy link
Contributor

This seems to be an issue in pip/setuptools that is triggered by this package. I don't have freebsd, so I can't help you.
Since python-zlib-ng is a drop-in replacement, it is technically redundant. If it is a dependency of a package you use, please request of them that they make it an optional requirement.

@LukaszMoskala
Copy link
Contributor Author

@ryanwinter I'm unable to reproduce this with Python 3.11.5 and pip 23.2.1 on FreeBSD 13.2.

You can look at this similar issue on stackoverflow.com

One thing that comes to mind is that FreeBSD 13.1 is no longer supported, and maybe upgrading to 13.2 or 14.0 would help.

Please provide output of:

freebsd-version -k #(on host when running in jail)
freebsd-version -u
pip --version
python --version

@precurse
Copy link

precurse commented Dec 12, 2023

Looks like when I try to compile the latest develop version, it's not adding the /usr/local/include/zlib-ng.h path when calling cc. If I have some time I'll try to fix, but wanted to mention it (FreeBSD 13.2-RELEASE in a Jail and Python311 venv):

 $ python3 setup.py install
/usr/home/hass/havenv_py11/lib/python3.11/site-packages/setuptools/dist.py:530: UserWarning: Normalizing '0.3.0-dev' to '0.3.0.dev0'
  warnings.warn(tmpl.format(**locals()))
running install
/usr/home/hass/havenv_py11/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
/usr/home/hass/havenv_py11/lib/python3.11/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
  warnings.warn(
running bdist_egg
running egg_info
writing src/zlib_ng.egg-info/PKG-INFO
writing dependency_links to src/zlib_ng.egg-info/dependency_links.txt
writing top-level names to src/zlib_ng.egg-info/top_level.txt
reading manifest file 'src/zlib_ng.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no directories found matching 'src/zlib_ng/zlib-ng'
adding license file 'LICENSE'
writing manifest file 'src/zlib_ng.egg-info/SOURCES.txt'
installing library code to build/bdist.freebsd-13.2-RELEASE-amd64/egg
running install_lib
running build_py
running build_ext
CMake Warning:
  Ignoring extra path from command line:

   "/tmp/tmpk9qr9w95"


CMake Error: The source directory "/tmp/tmpk9qr9w95" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
Error: could not load cache
building 'zlib_ng.zlib_ng' extension
cc -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -fPIC -I/tmp/tmpk9qr9w95 -I/usr/home/hass/havenv_py11/include -I/usr/local/include/python3.11 -c src/zlib_ng/zlib_ngmodule.c -o build/temp.freebsd-13.2-RELEASE-amd64-cpython-311/src/zlib_ng/zlib_ngmodule.o
src/zlib_ng/zlib_ngmodule.c:8:10: fatal error: 'zlib-ng.h' file not found
#include "zlib-ng.h"
         ^~~~~~~~~~~
1 error generated.
error: command '/usr/bin/cc' failed with exit code 1

It does exist:

# ls /usr/local/include/zlib-ng.h
/usr/local/include/zlib-ng.h

I tried adding /usr/local to possible_prefixes, but that didn't seem to help.

@LukaszMoskala
Copy link
Contributor Author

You probably need to do git submodule init and git submodule update as it expects it to be inside python-zlib-ng repo (which makes sense as you probably want to pin zlib-ng version to specific python-zlib-ng version)

@precurse
Copy link

That worked. Thanks!

@ryanwinter
Copy link

ryanwinter commented Dec 14, 2023

Thanks @LukaszMoskala,

I'm running 13.2 right now in the jail, however the host is still on 13.1.

I actually found the cause, the os.sched_getaffinity(0) in the setup.py is throwing an error (or maybe returning 0?). I just commented it out to use the os.cpu_count() instead. Not sure why this isn't working, didnt have time to investigate :(

@ryanwinter I'm unable to reproduce this with Python 3.11.5 and pip 23.2.1 on FreeBSD 13.2.

You can look at this similar issue on stackoverflow.com

One thing that comes to mind is that FreeBSD 13.1 is no longer supported, and maybe upgrading to 13.2 or 14.0 would help.

Please provide output of:

freebsd-version -k #(on host when running in jail)
freebsd-version -u
pip --version
python --version

@LukaszMoskala
Copy link
Contributor Author

I'm running 13.2 right now in the jail, however the host is still on 13.1.

That's probably your problem, as userspace should never be on higher version than kernel. Looks like some kernel API has changed (or been added in 13.2), you have application that uses it (python compiled for 13.2) but your kernel does not support it.

I'm always updating my host before updating my jails to solve that problem. So far having newer kernel (13.2) with older userspace (13.1 and 13.2) does not seem to cause any problems like that.

Anyway, I think that's outside of scope of this issue

@rhpvorderman
Copy link
Contributor

BSD installation enabled now on 0.3.0

@pixel8383
Copy link

If I try to install the package (pip install zlib-ng --no-cache-dir), I get the same error as @ryanwinter (OverflowError: could not allocate a large enough CPU set).

I am running a jail (13.2-RELEASE-p6) on an host running FreeBSD 13.1-RELEASE-p9. The jail runs with pip 23.3.2 and Python 3.11.5.

Could you please ELI5 how can I fix the issue?

Thank you!

@LukaszMoskala
Copy link
Contributor Author

LukaszMoskala commented Jan 9, 2024 via email

@pixel8383
Copy link

Yep @LukaszMoskala, I got it and, for sure, I will keep it in mind for the future but I think I cannot rollback the jail version easily. As far as I understood, I can workaround this forcing a git update somewhere, am I wrong?

@LukaszMoskala
Copy link
Contributor Author

LukaszMoskala commented Jan 9, 2024 via email

@pixel8383
Copy link

Basically you can either reinstall jail to older version (or rollback snapshot/backup if you have) or upgrade kernel on host. I don't know about any other way unfortunately.

Unfortunately I have no snapshot to roolback to (for different reasons I had to delete them). I guess I will have to wait until I can upgrade the kernel on the host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants