Open Discussion: How to handle package installations for duplicate but differing requirements

[raman325]

One limitation with the current implementation for package installation is that two different requirements.txt can list the same package with different version requirements that may not be compatible (e.g. pkg==0.0.1 and pkg==1.0.0), and the package will be installed multiple times in the order that it was parsed which could lead to potential issues down the line. What should the right behavior be? There are a couple of decisions to make:

1. Do we try to resolve this at all or do we let it happen with a caveat in the documentation?
2. If we try to resolve this, we can make a best effort to find a version that matches all of the requirements, but if we can't, does the newer version or the older version take precedence?

The step to find a version that matches all of the requirements would normally be achieved by passing a constraints file to pip, but we are using Home Assistant's installation mechanism which uses its own constraints file, so that's not a valid option for us. We would have to essentially implement our own version of this, which we may be able to use pkg_resources to do, I just haven't had a chance to look into it yet, and I wanted to get your thoughts @craigbarratt before I spend any time on this

[raman325]

One other thing to note - this will only be an issue in a scenario where the package doesn't already exist and we process two requirements files with the same package for installation in the same cycle. Once the package has been installed, future conflicts will be discarded based on our current logic, so maybe this isn't worth trying to implement a solution for.

[dlashua]

I'm not sure exactly what information you have available to make decisions but, if possible...

If Home Assistant (or any other custom_component, etc) requires a certain version of a package, that should be the version that gets installed. We should never change something Home Assistant has already done. This will help ensure that Home Assistant doesn't break as a result of package installation.

If Home Assistant doesn't care about a certain package, then we should install the highest version number requested by any pyscript for that package and, if it break, it breaks. The user can sort it out. We should be willing to downgrade a package, though. So, if I install version 5 of something and it breaks something else that asks for version 3 and then I realize I can do just fine with version 3 so I change the requirements, on the next pass it should see that the highest requested version is now 3 and attempt to downgrade to that.

The docs should advise users to pin to the least specific version that will still work in order to provide flexibility.

[raman325]

The code already prevents pyscript from overwriting an existing package -> HomeAssistant will always install over a version if the existing version mismatches what it needs, whether that's for core functionality or custom_components.

Regarding your second point, sounds like we should persist packages installed via pyscript so that we have a way to downgrade in the future (we only want to downgrade if pyscript installed the package, not if anyone else did).

I think advising users to pin to a specific version, and maybe even only accepting pinned versions for installation, is the best way to handle this. It will make conflict resolution much easier to solve.

[dlashua]

I guess the problem with pinning to a very specific version is that everything has to be exactly the same. When you maintain all of your own code, this is fine. But, when you're using pyscript apps written by others, this means if you add a pyscript that requires version 0.0.2 of a module, and you already have a pyscript that requires 0.0.1 (even though it might work just fine with 0.0.2) we have an unresolvable situation. By asking pyscript authors to indicate the minimum least specific version of a required package, yes, it might break just because the package API changed and the requested version was not specific enough. But, it'd be broken either way, in that case.

For downgrades, if there's a way we can know that Home Assistant installed a package we won't need to track what pyscript has installed. Otherwise, yeah, we'd need to track it to know it's okay to downgrade, but could run into a situation where Home Assistant adds a required package that we've already installed previously through pyscript. Thus resulting in pyscript thinking it's okay to downgrade when it's really not.

[raman325]

Sure, but if we take what you suggested above (the higher version wins) this conflict wouldn't be an issue, we would just install 0.0.2. The reason dependency resolution becomes easier with pinning is that I don't have to resolve the semantics of the requirement to figure out the right version install, I can be dumb and just say higher wins.

I can't think of a clean way to know what packages Home Assistant has installed. I'd have to read the requirements in to figure that out which I'd have to get through the filesystem because HA doesn't expose this. I think it'll be easier to track version installs internally. I can detect when Home Assistant (or any other system) installs over pyscript if I track the version I expect to be there.

[dlashua]

That seems reasonable.

[raman325]

No testing yet but at a high level this code master...raman325:requirement_mismatches_and_installed_pkg_storage would:

1. Track packages installed by pyscript.
2. Require version pinning.
3. Always defer to the highest version defined in requirements.txt.
4. Install any package version that is either not already installed or was installed by pyscript (and this handles if it was originally installed by pyscript but another actor came in after the fact and changed the version)

There is one scenario it doesn't cover - I want package version 1.0.0 that isn't previously installed. HA then updates and starts using package version 1.0.0 for a core component. I now want to use version 1.5.0 -> because the current version installed is one that I think was installed by me, I update it, potentially breaking the core component. Seems like a pretty small edge case though.

I will still need to add tests but leaving this here for an initial review

[raman325]

Updated with tests, going to go ahead and submit this for PR for further discussion