Nix Cache? #84

nomeata · 2019-03-07T11:02:14Z

I am playing around with haskell.nix right now, and while I am watching nix-tools build, I wonder if there is no nix cache with the binaries that I could use?

(That would probably also require you to pin nixpkgs to a specific version – but wouldn’t that be good in general, so that users will know which version of nixpkgs this project works best with, in particular with regard to fragile things like cross-compilation?)

The text was updated successfully, but these errors were encountered:

nomeata · 2019-03-07T11:05:23Z

Ah, I found https://github.com/input-output-hk/cardano-sl/blob/develop/docs/nix.md and it actually fetched some stuff (but not a lot – nixpkgs mismatch?). So consider this a suggestion to add these instructions to the documentation.

angerman · 2019-03-07T13:44:35Z

There is a cachix instance at nix-tools.cachix.org which can be use with cachix use nix-tools.
This is usually populated by the travis job. See https://github.com/input-output-hk/nix-tools/blob/master/.travis.yml

However the 50min limit means it needs perpetual manual intervention :-/ It also only builds agains nixos-18.09.

In general it doesn't matter against what nixpkgs nix-tools is built, and as such it might make sense to try to just get nix-tools as a regular package into nixpkgs; and some more frequent updates of the package to hackage.

nix-tools is just your stack.yaml, cabal.project, *.cabal to nix translator. You can even build it perfectly fine outside of nix. Use it and consume the results via haskell.nix.

We have an abstraction even one layer above haskell.nix, which you might want to look into: https://github.com/input-output-hk/iohk-nix; this contains common utility functionality we use at IOHK. If you are after windows cross compilation, you will find a lot of patches in there related to ghc, cabal and a few other packages.

Using iohk-nix you can reduce the default.nix and release.nix to a few lines using the templates from iohk-nix as shown in cardano-chain here:

https://github.com/input-output-hk/cardano-chain/blob/master/default.nix
https://github.com/input-output-hk/cardano-chain/blob/master/release.nix

nomeata · 2019-03-07T14:14:27Z

Thanks for the explation!

So much layers of abstraction… hard to keep up. But if the necessary patches are in https://github.com/input-output-hk/iohk-nix I guess I won’t have a choice to look into that as well…

nomeata · 2019-03-07T14:53:21Z

While I managed to build it using a plain haskell.nix setup, using iohk-nix, as I did here (which shows traces of cargo-culting) results it the build trying to build ghc, and that fails while patching it:

~/projekte/tip-toi-reveng $ nix-build nix -A nix-tools.exes.tttool
…
patching file libraries/Cabal/cabal-install/Distribution/Client/SetupWrapper.hs
patching file libraries/Cabal/cabal-testsuite/PackageTests/InternalLibraries/Executable/setup-static.test.hs
applying patch /nix/store/aqscn9ldnzj45hryfmhq670m83zwnc9j-ghc-8.4.3-Cabal2201-SMP-test-fix.patch
patching file libraries/Cabal/Cabal/tests/HackageTests.hs
applying patch /nix/store/rcgbp1587w8qlxcqfcwzifi44xmz91n1-outputtable-assert-8.4.patch
patching file compiler/typecheck/TcHsSyn.hs
patching file compiler/utils/Outputable.hs
applying patch /nix/store/pshjs4zbv23wyyakbpkrn5w7r7kq8xhz-iserv-proxy-cleanup.patch
patching file configure.ac
Hunk #1 FAILED at 1334.
1 out of 1 hunk FAILED -- saving rejects to file configure.ac.rej
patching file docs/users_guide/ghci.rst
Hunk #1 succeeded at 3243 (offset -44 lines).
patching file libraries/libiserv/libiserv.cabal.in
patching file libraries/libiserv/src/Lib.hs
patching file libraries/libiserv/src/Remote/Slave.hs
Hunk #4 FAILED at 123.
1 out of 4 hunks FAILED -- saving rejects to file libraries/libiserv/src/Remote/Slave.hs.rej
patching file utils/iserv-proxy/Makefile
patching file utils/iserv-proxy/iserv-proxy.cabal
Reversed (or previously applied) patch detected!  Assume -R? [n]
Apply anyway? [n]
Skipping patch.
1 out of 1 hunk ignored -- saving rejects to file utils/iserv-proxy/iserv-proxy.cabal.rej
patching file utils/iserv-proxy/iserv-proxy.cabal.in
patching file utils/iserv-proxy/src/Main.hs
patching file utils/iserv/iserv.cabal.in
patching file utils/remote-iserv/Makefile
patching file utils/remote-iserv/Setup.hs
patching file utils/remote-iserv/ghc.mk
patching file utils/remote-iserv/remote-iserv.cabal.in
patching file utils/remote-iserv/src/Cli.hs
builder for '/nix/store/j6m77ilz08bsdhs8cba9i0mhs4lw0vak-ghc-8.4.4.drv' failed with exit code 1
cannot build derivation '/nix/store/144mx0wiw2xmribkfxmp6qj2blri6jb9-tttool-1.8-exe-tttool.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/3k86qldz7bch11xb3g1d047v5xggx9jm-tttool-exes.drv': 1 dependencies couldn't be built
error: build of '/nix/store/3k86qldz7bch11xb3g1d047v5xggx9jm-tttool-exes.drv' failed

angerman · 2019-03-08T01:24:31Z

I suspect your iohk-nix checkout is too old. Bumping it past input-output-hk/iohk-nix@ed41749 in https://github.com/entropia/tip-toi-reveng/blob/5c7495df222e270c30bd81f027da8ffb6f63b5a2/nix/iohk-nix.json should do the trick.

angerman · 2019-03-08T01:27:05Z

So much layers of abstraction… hard to keep up. But if the necessary patches are in https://github.com/input-output-hk/iohk-nix I guess I won’t have a choice to look into that as well…

This is a real concern. With the amount of patches required for trivial cross compilation of haskell projects to windows, this is a sad necessity. Once all is upstreamed, the iohk-nix layer could be omitted.

nomeata · 2019-03-08T10:20:01Z

I suspect your iohk-nix checkout is too old.

oh, sorry, should have checked. Too much copy’n’pasting… now it fetched ghc-8.4.4 from your hydra, so that looks more promising.

nomeata · 2019-03-08T10:36:06Z

Ok, I can build it “normally”. Now I am really excited about the cross build … but I just can’t figure what the attribute it that builds it :-/

My best bet, setting the crossSystem argument as in

nix-build nix -A nix-tools.exes.tttool -j8 --arg crossSystem '{ config = "x86_64-pc-mingw32"; }'

does not seem to affect the output.

nomeata · 2019-03-08T13:34:30Z

With more inspiration from https://github.com/input-output-hk/cardano-chain/blob/master/release.nix I wrote this release file in entropia/tip-toi-reveng@b63408c:

let
  localLib = import ./lib.nix;
in
localLib.nix-tools.release-nix {
  package-set-path = ./.;
  packages = [ "tttool" ];
  required-targets = jobs: [
    jobs.nix-tools.exes.tttool.x86_64-linux
    # windows cross compilation targets
    jobs.nix-tools.exes.x86_64-pc-mingw32-tttool.x86_64-linux
];
}

and now get an error message like this:

error: attribute 'Win32' missing, at /nix/store/16sxhsfhvya2yvn6xdjqrz5c6m9w8yp1-hackage-exprs-source/hackage/directory-1.3.1.5-r0-32305179022172a41fc74b83c726a1e4385725dee53ccb1fb869d000013a3788.nix:24:19
(use '--show-trace' to show detailed location information)

Any ideas at this point?

angerman · 2019-03-08T14:01:14Z

@nomeata the issue is that your plan.json is built against linux (I assume), and as such misses the Win32 package.

You can do two things (I think), run cabal new-configure and set windows as the OS, or just add the missing packages back in:

https://github.com/entropia/tip-toi-reveng/blob/b63408c1a5ef7d5996df8e26651f5834d3d0058f/nix/pkgs.nix#L24-L27

adding
(hackage: { Win32 = hackage.Win32."2.6.2.0".revisions.default; })

to yield

pkg-def-overlays = [
      { tttool = ./tttool.nix;
        HPDF = ./HPDF.nix;
      }
      (hackage: { Win32 = hackage.Win32."2.6.2.0".revisions.default; })
];
would help with that, I hope.

nomeata · 2019-03-08T14:32:52Z

Whooho,it’s building something :-D

nomeata · 2019-03-08T17:06:37Z

So, it cross-built ghc and a bunch of Haskell libraries, and got stuck at conduit:

building '/nix/store/pwqr3vc0bh317sbfvq6va9rrrn15z1yb-conduit-1.3.0.3-lib-conduit-x86_64-pc-mingw32.drv'...
checking whether the C compiler works... yes
checking for C compiler default output file name... a.exe
[1 of 2] Compiling System.Environment.Executable.Win32 ( System/Environment/Executable/Win32.hs, dist/build/System/Environment/Executable/Win32.o )
checking for suffix of executables... .exe

System/Environment/Executable/Win32.hs:25:1: warning: [-Wunsupported-calling-conventions]
    • the 'stdcall' calling convention is unsupported on this platform,
      treating as ccall
    • When checking declaration:
        foreign import stdcall unsafe "Windows.h GetModuleFileNameW" c_GetModuleFileNameW
          :: HMODULE -> Ptr CWchar -> Word32 -> IO Word32
   |
25 | foreign import stdcall unsafe "Windows.h GetModuleFileNameW" c_GetModuleFileNameW
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...
unpacking sources
unpacking source archive /nix/store/h28a0knml06jc52psjjld17mzpaj22g7-conduit-1.3.0.3.tar.gz
source root is conduit-1.3.0.3
setting SOURCE_DATE_EPOCH to timestamp 1526371772 of file conduit-1.3.0.3/test/Spec.hs
patching sources
configuring
Configure flags:
--prefix=/nix/store/76cgvnw6l8bl30gcq6srdld0ix2j0rpx-conduit-1.3.0.3-lib-conduit-x86_64-pc-mingw32 lib:conduit --package-db=clear --package-db=/nix/store/0al24n0d1g1k4jmh48lbdm4wsrkaw1zs-conduit-1.3.0.3-lib-conduit-config-x86_64-pc-mingw32/package.conf.d --with-ghc=x86_64-pc-mingw32-ghc --with-ghc-pkg=x86_64-pc-mingw32-ghc-pkg --with-hsc2hs=x86_64-pc-mingw32-hsc2hs --with-gcc=x86_64-pc-mingw32-cc --with-ld=x86_64-pc-mingw32-ld --with-ar=x86_64-pc-mingw32-ar --with-strip=x86_64-pc-mingw32-strip --enable-executable-stripping --enable-library-stripping --hsc2hs-option=--cross-compile --hsc2hs-option=--via-asm
[2 of 2] Compiling System.Environment.Executable ( System/Environment/Executable.hs, dist/build/System/Environment/Executable.o )
checking whether we are cross compiling... yes
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /nix/store/mjrw6436gnxfwadswsvf8v4bpq6b2cj9-x86_64-pc-mingw32-stage-final-gcc-debug-wrapper-7.3.0/bin/x86_64-pc-mingw32-cc accepts -g... yes
installing
checking for /nix/store/mjrw6436gnxfwadswsvf8v4bpq6b2cj9-x86_64-pc-mingw32-stage-final-gcc-debug-wrapper-7.3.0/bin/x86_64-pc-mingw32-cc option to accept ISO C89... none needed
Installing library in /nix/store/hr31zrmh0hr3f3kq8sr0zjfsg1299c8d-executable-path-0.0.3.1-lib-executable-path-x86_64-pc-mingw32/lib/x86_64-windows-ghc-8.4.4/executable-path-0.0.3.1-GagQtx0OywH7hgXaad4APP
checking how to run the C preprocessor... /nix/store/mjrw6436gnxfwadswsvf8v4bpq6b2cj9-x86_64-pc-mingw32-stage-final-gcc-debug-wrapper-7.3.0/bin/x86_64-pc-mingw32-cc -E
Configuring library for conduit-1.3.0.3..
Loaded package environment from /nix/store/0al24n0d1g1k4jmh48lbdm4wsrkaw1zs-conduit-1.3.0.3-lib-conduit-config-x86_64-pc-mingw32/ghc-environment
checking for grep that handles long lines and -e... /nix/store/fr6apsdp46f9m1290gxgfmhfc4rpmcjw-gnugrep-3.1/bin/grep
checking for egrep... /nix/store/fr6apsdp46f9m1290gxgfmhfc4rpmcjw-gnugrep-3.1/bin/grep -E
checking for ANSI C header files... yes
post-installation fixup
patching script interpreter paths in /nix/store/hr31zrmh0hr3f3kq8sr0zjfsg1299c8d-executable-path-0.0.3.1-lib-executable-path-x86_64-pc-mingw32
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
building
checking for stdint.h... yes
Preprocessing library for conduit-1.3.0.3..
checking for unistd.h... yes
Win32File.hsc:34:10: fatal error: Share.h: No such file or directory
compilation terminated.
compilation failed
builder for '/nix/store/pwqr3vc0bh317sbfvq6va9rrrn15z1yb-conduit-1.3.0.3-lib-conduit-x86_64-pc-mingw32.drv' failed with exit code 1

nomeata · 2019-03-08T17:12:38Z

According to https://stackoverflow.com/questions/19918173/cant-install-conduit-on-windows, people on Windows solve this by using MinGW instead of cygwin shell… not sure if and how that insight translates into the nix world.

conduit-1.3.1 has the same problem.

Maybe it ever cross-built, and it just happens to be a package that is not used at IOHK, so nobody bothered to investigate?

I depend on conduit via yaml, this project really needs yaml, and yaml really needs conduit, so I can’t just patch it out.

nomeata · 2019-03-08T17:53:59Z

Ha, lower-casing the imports does the trick!

nomeata · 2019-03-08T18:03:07Z

Success!

~/projekte/tip-toi-reveng $ WINEPREFIX=~/.wine64 wine64 /nix/store/c0nqyyd9wz43an5jwsvhpq6v912hcmpx-tttool-exes-x86_64-pc-mingw32/bin/tttool.exe
0009:fixme:process:GetNumaHighestNodeNumber (0x1c88798): semi-stub
Missing: COMMAND

Usage: tttool.exe [-t|--transscript FILE] [--code-dim W[xH]] [--dpi DPI]
                  [--pixel-size N] [-f|--image-format Format] COMMAND
  tttool-1.8 -- The swiss army knife for the Tiptoi hacker

Thanks a lot for your help!

nomeata · 2019-03-08T18:51:43Z

Next goal: Static builds for linux. Is that something that is supported by the IOHK tooling as well? Is it worth trying to build with { crossSystem = localLib.systems.examples.musl64;};?
Or do you simply distributed non-static binaries, or glibc-static-binaries?

hackage-to-nix tool operates on hard-coded path to global Hackage database. This commit allows to optionally specify alternative Hackage 01-index.tar file, along with Hackage url. This url is then hardcoded into generated nix expressions, to make sure that their source can be fetched when building.

nomeata mentioned this issue Mar 8, 2019

Use lower-case imports (better for cross-compilation) snoyberg/conduit#408

Merged

nomeata closed this as completed Mar 8, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Nix Cache? #84

Nix Cache? #84

nomeata commented Mar 7, 2019

nomeata commented Mar 7, 2019 •

edited

Loading

Uh oh!

angerman commented Mar 7, 2019

Uh oh!

nomeata commented Mar 7, 2019

Uh oh!

nomeata commented Mar 7, 2019 •

edited

Loading

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019 •

edited

Loading

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

Nix Cache? #84

Nix Cache? #84

Comments

nomeata commented Mar 7, 2019

nomeata commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman commented Mar 7, 2019

Uh oh!

nomeata commented Mar 7, 2019

Uh oh!

nomeata commented Mar 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

angerman commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 8, 2019

Uh oh!

nomeata commented Mar 7, 2019 •

edited

Loading

nomeata commented Mar 7, 2019 •

edited

Loading

nomeata commented Mar 8, 2019 •

edited

Loading