cabal-install seems to pick wrong library versions in Template Haskell splices if there are multiple versions in the store

[patrickt]

Describe the bug
In the course of developing https://github.com/github/semantic and https://github.com/tree-sitter/haskell-tree-sitter, several of us have encountered a mysterious bug in which a library dependency, during execution of a Template Haskell splice, appears to link in an incorrect library version if there are multiple versions of this library present in the store. This has happened across a number of cabal-install versions; I use v3.0.

Behavior

After some time spent developing in a given checkout of semantic, the package store gets into a state where invoking cabal build triggers a rebuild of our dependencies:

Build profile: -w ghc-8.6.5 -O0
In order, the following will be built (use -v for more details):
 - lingo-0.3.0.0 (lib:lingo) (requires build)
 - proto-lens-jsonpb-0.1.0.0 (lib) (configuration changed)
 - semantic-core-0.0.0.0 (lib) (file src/Core/Core.hs changed)
 - tree-sitter-json-0.3.0.0 (lib) (requires build)
 - tree-sitter-java-0.4.0.0 (lib) (requires build)
 - tree-sitter-php-0.2.0.1 (lib) (requires build)
 - tree-sitter-ruby-0.2.0.1 (lib) (requires build)
 - tree-sitter-typescript-0.2.1.1 (lib) (requires build)
 - semantic-python-0.0.0.0 (lib) (dependency rebuilt)
 - semantic-json-0.0.0.0 (lib) (dependency rebuilt)
 - semantic-java-0.0.0.0 (lib) (dependency rebuilt)
 - semantic-0.9.0.0 (lib) (configuration changed)
 - semantic-0.9.0.0 (exe:semantic) (configuration changed)

Things start to go awry when we compile the tree-sitter-* subpackages.

Configuring library for tree-sitter-json-0.3.0.0..
Warning:
    This package indirectly depends on multiple versions of the same package. This is very likely to cause a compile failure.
      package fused-effects (fsd-ffcts-0.5.0.1-9a31efa4) requires MndRndm-0.5.1.1-8f6cd463
      package fused-effects (fsd-ffcts-0.5.0.1-df5a4d06) requires MndRndm-0.5.1.1-d24d69b0
      package tree-sitter (tr-sttr-0.5.0.0-2ca28879) requires sn-1.4.6.0-14021fe2
      package semantic-source (smntc-src-0.0.0.1-9469622a) requires sn-1.4.6.0-14021fe2

Each of these subpackages consists of a library and an internal library. The internal library contains generated C code, and the library itself runs a Template Haskell splice (defined in the base tree-sitter library) to generate and expose data types based on the symbols defined by this internal library. (I understand that it may be unconventional to do this level of work in a TH splice, including FFI invocations, but we have quite specific requirements.) At this point, that TH splice fails with an error suggesting that the wrong version of a library has been linked in:

[1 of 2] Compiling TreeSitter.JSON  ( TreeSitter/JSON.hs, dist/build/TreeSitter/JSON.o )

TreeSitter/JSON.hs:15:37: error:
    • Couldn't match type ‘tree-sitter-0.5.0.0:TreeSitter.Language.Language’
                     with ‘Language’
      NB: ‘Language’
            is defined in ‘TreeSitter.Language’
                in package ‘tree-sitter-0.5.0.0’
          ‘tree-sitter-0.5.0.0:TreeSitter.Language.Language’
            is defined in ‘TreeSitter.Language’
                in package ‘tree-sitter-0.5.0.0’
      Expected type: GHC.Ptr.Ptr Language
        Actual type: GHC.Ptr.Ptr
                       tree-sitter-0.5.0.0:TreeSitter.Language.Language
    • In the second argument of ‘mkSymbolDatatype’, namely
        ‘tree_sitter_json’
      In the expression:
        mkSymbolDatatype (mkName "Grammar") tree_sitter_json
   |
15 | mkSymbolDatatype (mkName "Grammar") tree_sitter_json
   |                                     ^^^^^^^^^^^^^^^^

This repeats itself for all the tree-sitter-* packages. It seems to imply that there are two tree-sitter packages in the store, and that the tree_sitter_json identifier being used here is linked to a different version than is this invocation of mkSymbolDatatype.

To Reproduce
I have not been able to figure out how to reproduce this behavior from a clean checkout. It tends to happen when ~/.cabal/store contains a great deal of duplicated packages: here’s a list of what’s present in .cabal/store that’s currently displaying this behavior. (Note that there are two tr-sttr entries; I presume that different ones are being picked at different phases. If it’s helpful, I’m happy to upload the entirety of my .cabal directory, though its size (4.5GB) is considerable. Removing dist-newstyle in the semantic folder doesn’t change the behavior; the only thing I’ve found that fixes it is nuking ~/.cabal/store and ~/.cabal/packages, which entails a bit of recompilation.

Expected behavior
This should finish building.

System information

• macOS (reproduced on 10.14 and 10.5)
• cabal version 3.0.0.0, ghc version 8.6.5.

Additional context
I realize that this is somewhat of an involved bug, so if this is a consequence of some way in which we are using the tooling wrong, I’m happy to accept any suggestions as to how we can better address this, and to provide any other information that you might need.

[phadej]

I'm quite sure the root cause is the same (mystical one) as in #6211 and #5782

However, we failed to find out how to reproduce that (this ?) issue reliably, so the best we can do now is "wipe the store, start over".

Correction: except that grayjay got a repro-case in #5782, I haven't tried it yet myself though.

[patrickt]

We’re seeing this on our CI servers now. 😭 We’ve worked around this locally: https://github.com/github/semantic/blob/c78529d06187ee09ccf5dc4b5f6fb47788403eab/script/fix-broken-cabal-store

[robrix]

@phadej is there reason to believe that this would affect public sub-libraries as well as private internal libraries?

[jneira]

@patrickt we are woking around using a separate store-dir for builds and install (but you will need more cache if you want to add the extra store-dir): #6211 (comment)

[Mikolaj]

Closing as duplicate of #5782, according to Oleg's hunch.

@patrickt: could you kindly try to reproduce this problem with a current stable cabal? We are guessing in #5782 (comment) this issue is fixed in new stable cabals, but we'd love as much confirmation as we can get.

@robrix: yes, I guess it was likely to affect both private and public internal libraries. Let's hope it does not longer.