Please, fix HNix build inside Nixpkgs

[Anton-Latukha]

Monitored for package, 0.9.0 hit the Nixpkgs, and in broken state: https://hydra.nixos.org/job/nixpkgs/trunk/haskellPackages.hnix.x86_64-linux#tabs-status

At my last HNix-related PR into Nixpkgs, I did not wanted generateOptparseApplicativeCompletion to be enabled in the Nixpkgs (because we were not testing for completion support in our CI back then), but for me and maintainer there it seemed as it works.

In fact, after the merge, it is a fact that it does not work: https://hydra.nixos.org/build/122819514/nixlog/1

The reason is just, and in fact is the same reason why project can not pass its own tests in Nixpkgs:

  Eval comparison tests
    ind-string-17.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-16.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-15.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-14.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-13.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-12.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-11.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-10.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-09.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-08.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-07.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-06.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-05.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-04.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-03.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-02.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    ind-string-01.nix:                                               FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    builtins.string.store.nix:                                       FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    builtins.split-04.nix:                                           FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    builtins.split-03.nix:                                           FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    builtins.split-02.nix:                                           FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)
    builtins.split-01.nix:                                           FAIL
      Exception: nix-instantiate: readCreateProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Because in Nixpkgs there is no Nix submodule. Executable would not be built until the project would become independent from Nix submodule use, so currently it would be just a library in Nixpkgs.

[Anton-Latukha]

I address this report in NixOS/nixpkgs#91312.

At this point in time it would be a couple of days before it would be merged.

[sorki]

Maybe we can add an override like in https://github.com/haskell-nix/hnix/blob/master/default.nix#L190 - probably not the right way but I'm not sure how these (librarySystemDepends, testSystemDepends,..) are discovered for Haskell package set in NixOS but only few have such overrides defined, rest seem to be automatically populated somehow or maybe I'm just missing a source for that.

[Anton-Latukha]

For me it seems reverse to logic, since those would be are our own efforts put to dodge our own project bugs downstream, while those efforts effectively to be put into the project itself addressing the real cause of it.

Also, I do not think the PR with such workarounds is a cdepillabout permission bar. It would be quite hard to get his allowance to drag-in something like this.

Since we have tests and completion testing in CI that tests inside current Nixpkgs - it is OK to have it like this in Nixpkgs for a couple of month. The most important thing that we test it, and that working library is provided to the community and dependent projects.

We just need to get HNix into a great shape - that is our goal. At first, HNix should not depend on Nix, it is an oxymoron currently to have it this way. The project would be able to call itself reaching some initially stated goal when it at least is able to self-boot, before that - it is basically a library to other projects in which we not sure, because project can not self-test with own tool with Nixpkgs in the current state.

[Anton-Latukha]

Also the Nix should be provided in the proper ./data/nix path place for its source to be loaded by the project.

[Anton-Latukha]

I do not know the all angles of the scope of this issue. But since I see the main cause and its resolution that is aligned with major line of the project - that is sufficient for me. If we missed something, and there is a proper good way to workaround this so it works in Nixpkgs before we address it properly, happy to listen, hear and understand it better.

[Anton-Latukha]

Ok, yes, we missed something, and so I was wrong about the direct cause of this one.

In fact - the reason is that we use new Haskelline, so as you would now remember - we disabled the executable builds for older versions.

malahorns NixOS/nixpkgs#91251 addresses it better.

But overall. We are those who know the full picture. All these are partially working solutions to the problem, the executable from the Nixpkgs still would not evaluate the Nix code properly, just like tests suite fails inside Nixpkgs due to the absence of Nix submodule files.

[sorki]

For me it seems reverse to logic, since those would be are our own efforts put to dodge our own project bugs downstream, while those efforts effectively to be put into the project itself addressing the real cause of it.

Also, I do not think the PR with such workarounds is a cdepillabout permission bar. It would be quite hard to get his allowance to drag-in something like this.

Yup that's why I want to figure out how these native (system) deps are populated automatically - will ask peti during the Fridays stream I guess.

Since we have tests and completion testing in CI that tests inside current Nixpkgs - it is OK to have it like this in Nixpkgs for a couple of month. The most important thing that we test it, and that working library is provided to the community and dependent projects.

I agree!

We just need to get HNix into a great shape - that is our goal. At first, HNix should not depend on Nix, it is an oxymoron currently to have it this way. The project would be able to call itself reaching some initially stated goal when it at least is able to self-boot, before that - it is basically a library to other projects in which we not sure, because project can not self-test with own tool with Nixpkgs in the current state.

+1 - being able to bootstrap and build hnix with hnix will be a major step forward.

Thanks for fixing these, I'm going to fix hnix-store-remote and update-nix-fetchgit next.

To follow up on later comments there's also a problem with running the testsuite using systems Nix due to the requirement of the /nix/store - I work around this in hnix-store-remote by creating ad-hoc stores for testing in /tmp and running tests in namespaces so they won't pollute your systems store - I will try to refactor the harness a bit and make it a separate package so it can be re-used outside of -remote. It currently also suffers from the same issue as the original report as it needs nix-daemon to be able to test remote protocol against it.

[Anton-Latukha]

Well, the Nix installer had rudimentary hardcode abilities (should be able to) to switch setup from /nix/store into other path user wants. I do not know if Nix accepts those abilities.

I doubt that it works, Nix version and installer in URL was not updated for 1-2 years or so.

Now I know that macOS blocks the creation of /* paths, so they probably would be quick to address that ability. It is funny how much people care about complying proprietary macOS demands, but not about their own Free Software ecosystem and Linux distro relatives.

---

Nix store can not be polluted, it is the idea of it to be a central package build storage on the system, and it is easy to garbage collect it.

The default approach is the other way around - through configuration to setup fixed points, like to link projects and so their GHCs with lists of packages, so they detected as needed closures and not garbage collected, and everything else is cleaned-up.

[Anton-Latukha]

I was looking into a crystal ball in #641 (comment)

Since Nix upstream made a new release or Nix installer, at last. And it seems that it uses macOS originated installer code.

Well, while they would debug the new Nix installer and rebuild all closures, it broke our CI for the nearest days while Hydra CI would do rebuilds of everything and installer release and changes would be sorted-out.

[Anton-Latukha]

NixOS/nixpkgs#91251 that addresses the report is merged.

Case closed.

But lets keep it open till haskell-updates merges into master and build arrives. It is somewhere over the weekend.

[sorki]

Was looking at this and it is possible to add

build-tool-depends: nix:nix

to cabal file (for tests) which results in testToolDepends = [ nix ]; in cabal2nix output.

But it doesn't seem to change anything for me as tests are passing even without it (via nix-build and enabled sandbox).

For the reference https://cabal.readthedocs.io/en/stable/developing-packages.html?highlight=build-tool-depends#pkg-field-build-tool-depends