Skip to content

Packaging as namespace packages #13

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 34 commits into from
Jun 21, 2024
Merged

Packaging as namespace packages #13

merged 34 commits into from
Jun 21, 2024

Conversation

untereiner
Copy link

No description provided.

@untereiner untereiner requested a review from cssherman March 13, 2024 13:38
@untereiner untereiner self-assigned this Mar 13, 2024
@untereiner untereiner linked an issue Mar 13, 2024 that may be closed by this pull request
@untereiner
Copy link
Author

untereiner commented Mar 13, 2024

@cssherman I have finished the work for this PR
Also:

  • would it be possible to rename geosPythonPackages into geos-python ?
  • afterwards I would like to publish this packages to pypi

@cssherman
Copy link
Collaborator

@cssherman I have finished the work for this PR Also:

* would it be possible to rename `geosPythonPackages` into `geos-python` ?

* afterwards I would like to publish this packages to pypi

I'll start taking a look! I don't see any issues with the name change

docs/conf.py Outdated
@@ -18,7 +18,7 @@

# Add python modules to be documented
python_root = '..'
python_modules = ( 'geosx_mesh_tools_package', 'geosx_xml_tools_package', 'geosx_mesh_doctor', 'geos_ats_package',
python_modules = ( 'geosx_mesh_tools_package', 'geosx_xml_tools_package', 'geosx_mesh_doctor', 'geos-ats_package',
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The package names, paths that are used in the python autodocs need to be updated. We can easily add a CI test to make sure docs build correctly before deployment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a CI workflow to check the documentation.

@@ -1,5 +1,5 @@
import numpy as np
from hdf5_wrapper import hdf5_wrapper as h5w
from geos.hdf5.wrapper import hdf5_wrapper as h5w
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given the name changes, shouldn't this be:

from geos.hdf5wrapper import wrapper as h5w

@cssherman
Copy link
Collaborator

Since we are currently only testing package installation in the CI, we need to do a bunch of manual tests to make sure that the changes work. Also, @untereiner - could you open up a testing branch in the main GEOS repository, following these instructions:

https://geosx-geosx.readthedocs-hosted.com/projects/geosx-geospythonpackages/en/latest/#development-debugging

@cssherman
Copy link
Collaborator

* afterwards I would like to publish this packages to pypi

Before we do this, we need to clarify the license information in these packages / the top-level repo. We also need to check with @rrsettgast and others to get approval to do so.

@cssherman
Copy link
Collaborator

@untereiner - I'm also happy to setup the pip repository. I'm already managing one for another project, so I'm comfortable with the tools/workflows.

@untereiner
Copy link
Author

untereiner commented Mar 26, 2024

Since we are currently only testing package installation in the CI, we need to do a bunch of manual tests to make sure that the changes work. Also, @untereiner - could you open up a testing branch in the main GEOS repository, following these instructions:

https://geosx-geosx.readthedocs-hosted.com/projects/geosx-geospythonpackages/en/latest/#development-debugging

The CI of the GEOS repo fails for this branch because I changed the names of the modules, Then, the integrated tests don't find the geos-ats module.

@untereiner
Copy link
Author

What are the failures ? pyproject.toml is standard

@cssherman
Copy link
Collaborator

What are the failures ? pyproject.toml is standard

Yes, but this is a pretty new standard. Using a version of pip that was only 2 versions old, this lead to silent errors that were very hard to debug. I don't think there is a way to enforce this, so we need to add some big warning boxes to the docs.

@untereiner
Copy link
Author

When you say two versions old, you start with python 3.12 ?
Each version of python is distributed with the last version (at that moment) of pip
Regarding the status of versions of python it should be a moving window (not too large) of allowed versions.
For example the year 2024 -> [3.9, 3.12]. then for 2015 -> [3.10, 3.13], etc.
This will enforce the min pip version allowed to build the project.
This can be stated in the documentation

@cssherman
Copy link
Collaborator

@untereiner - do we have regular users of mesh doctor that we can call on to test these changes? We don't have any integrated tests involving that package in the main repo (yet).

@cssherman
Copy link
Collaborator

@untereiner - Looking at these changes again, there were a handful of module name changes that added more depth that I like in the namespace. There were also a number of instances where the changes weren't reflected properly in test scripts, documentation, etc. I've done some additional work to address this in my recent commit.

@cssherman
Copy link
Collaborator

I've also fixed some doc build issues, and consolidated some of the older mesh conversion tools and the doctor into a single 'geos-mesh' package

@cssherman
Copy link
Collaborator

When you say two versions old, you start with python 3.12 ? Each version of python is distributed with the last version (at that moment) of pip Regarding the status of versions of python it should be a moving window (not too large) of allowed versions. For example the year 2024 -> [3.9, 3.12]. then for 2015 -> [3.10, 3.13], etc. This will enforce the min pip version allowed to build the project. This can be stated in the documentation

When you say two versions old, you start with python 3.12 ? Each version of python is distributed with the last version (at that moment) of pip Regarding the status of versions of python it should be a moving window (not too large) of allowed versions. For example the year 2024 -> [3.9, 3.12]. then for 2015 -> [3.10, 3.13], etc. This will enforce the min pip version allowed to build the project. This can be stated in the documentation

I meant two versions old of pip (obviously not an ideal situation, but this was on a CI machine that may not have been updated in a while). While pyproject.toml files have been standard for a while, pure-pyproject.toml projects (i.e.: no setup.cfg) only seem to work with very recent versions of pip.

And with respect to documentation, I agree that we should note this behavior. For most users though, we just need to include a python -m pip install --upgrade step in the instructions (most users are still going to use pip, regardless of our support for other tools).

For python version support, I think that we need to support all non-deprecated versions of python. Some end-users may not have full control of their environments and we should do our best to make their lives easier.

@cssherman
Copy link
Collaborator

@CusiniM - do you have any thoughts about the re-organization of these packages? The big change here is how they would be imported. For example, from geos_ats.test_builder import TestDeck would now be from geos.ats.test_builder import TestDeck

@CusiniM
Copy link
Collaborator

CusiniM commented Jun 7, 2024

@untereiner - do we have regular users of mesh doctor that we can call on to test these changes? We don't have any integrated tests involving that package in the main repo (yet).

@jhuang2601 and I can test this. However the installation was already not working.

@CusiniM
Copy link
Collaborator

CusiniM commented Jun 7, 2024

@CusiniM - do you have any thoughts about the re-organization of these packages? The big change here is how they would be imported. For example, from geos_ats.test_builder import TestDeck would now be from geos.ats.test_builder import TestDeck

I don't think this is an issue. However, is there a specific reason for creating all those nested folders?
almost feels like we could ship everything as a single package at this point with a single toml?

@cssherman
Copy link
Collaborator

@CusiniM - do you have any thoughts about the re-organization of these packages? The big change here is how they would be imported. For example, from geos_ats.test_builder import TestDeck would now be from geos.ats.test_builder import TestDeck

I don't think this is an issue. However, is there a specific reason for creating all those nested folders? almost feels like we could ship everything as a single package at this point with a single toml?

I haven't packaged something like this before, but from what I understand, this allows us to independently package components of a single larger module. Maybe this has benefits for simplifying testing?

@untereiner
Copy link
Author

They can be packaged, tested, distributed separately but they can work together easily because they share the same root. It’s a kind of separation of concerns.

@untereiner
Copy link
Author

@cssherman For python versions it means min 3.8 until 2024/10
Also, python is very dynamic (in many ways) then sticking to old versions seems to be a bad idea for me even if updating CI machines is painful.
@CusiniM Each package can have it's own set of dependencies. There aren't necessarily cross dependencies.
Then:

  • as a final user wanting to preprocess my meshes I can do pip install geos-mesh
  • in the CI it is a pip install geos-ats.

What can I do now to help move forward ?
For the packaging, the next would be to update the CI to publish the package to pypi (github doesn't have a python package registry)..

@cssherman
Copy link
Collaborator

@cssherman For python versions it means min 3.8 until 2024/10 Also, python is very dynamic (in many ways) then sticking to old versions seems to be a bad idea for me even if updating CI machines is painful. @CusiniM Each package can have it's own set of dependencies. There aren't necessarily cross dependencies. Then:

* as a final user wanting to preprocess my meshes I can do `pip install geos-mesh`

* in the CI it is a `pip install geos-ats`.

What can I do now to help move forward ? For the packaging, the next would be to update the CI to publish the package to pypi (github doesn't have a python package registry)..

I think that this is ready to move forward. I've added the associated PR in the main GEOS repo to the dashboard, with a note on the procedure to handle the merge. We also need to have a discussion with the larger GEOS community to give them an explanation of the changes, so they aren't caught off-guard.

And WRT python versions, I agree that in a perfect world, we should only need to target reasonably up-to-date versions of python. At least in short-term, I think we need to keep with the current strategy.

@untereiner
Copy link
Author

@cssherman great!
It will be the right time to also rename the repository from geosPythonPackages into a more pythonic style like geos-python.

@cssherman cssherman merged commit e5abdc8 into main Jun 21, 2024
31 checks passed
@untereiner untereiner deleted the feature/packaging branch June 24, 2024 08:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

packaging
3 participants