Excessive memory usage in monorepos with dozens/hundreds of cabal packages or modules

[jneira]

• There are several issues about memory consumption but it seems it is exacerbated in those contexts
• The large public codebase where hls works reliably is ghc but it has few packages
• Test performance of large monorepos is hard:
  • They usually are private
  • Setup a benchmark here would be pretty difficult
• So we need the feedback of users to improve perfomance in those scenarios
• Pinging the experts om the subject: @wz1000, @mpickering, @pepeiborra, @fendor
• I think @parsonsmatt has experienced this issue
• As a first step we should analyze where is the culprit, starting from hie-bios

[pepeiborra]

@jneira thanks for the summary. I don't have time to look into this but can share a few pointers:

• we have a benchmark example for this scenario: HLS itself. It's currently disabled due to slowness, can someone look into reenabling it?
  

  haskell-language-server/ghcide/bench/config.yaml

  Lines 30 to 40 in a800b9d

  	# Small but heavily multi-component example
  	# Disabled as it is far to slow. hie-bios >0.7.2 should help
  	# - name: HLS
  	# path: bench/example/HLS
  	# modules:
  	# - hls-plugin-api/src/Ide/Plugin/Config.hs
  	# - ghcide/src/Development/IDE/Plugin/CodeAction/ExactPrint.hs
  	# - ghcide/bench/hist/Main.hs
  	# - ghcide/bench/lib/Experiments/Types.hs
  	# - ghcide/test/exe/Main.hs
  	# - exe/Plugins.hs

• once we have a working benchmark, it should be quite easy to characterise the scenarios that leak memory using the array of experiments available. The benchmark suite produces heap graphs and can also produce heap profiles

• the most likely culprit is GHC. The Session code creates a cascade of GHC Sessions to support multi-component scenarios, and it's likely that we are holding onto too much data (and leaking some more). There are two paths forward then:

  • thin out the stuff we keep alive to reduce the overall footprint,
  • identify leaks in GHC and fix them upstream. Since it is possible they are already fixed in HEAD, this step should be performed using GHC 9.2 or newer

Good luck and happy hacking!

[jneira]

Many thanks for your insights, for completeness the pr about enabling the hls becnhmark example is #1420

[guibou]

I'm having access to one private monorepo with ~250 "packages" and more than 1000 modules which exhibits huge memory usage (we upgraded dev machines to 32 GB of RAM in order to use HLS).

I still have good contact with another company with a similar monorepo.

Both are using bazel as the build system and the flags are generated from bazel. Note that there is one "component" (i.e. on cradle: ... shell: call per module).

I can run every benchmarking you may suggest on this repo.

[parsonsmatt]

The Haskell codebase at Mercury is 232kloc in a single cabal package with 1,600+ modules currently. Build system is cabal with nix providing all dependencies.

A large number of packages probably contributes to slowdown, but even with a single package target, HLS is unusably slow for most of our developers, even those with 32GB of RAM.

[fendor]

HLS is unusably slow for most of our developers, even those with 32GB of RAM.

I think some plugins contribute to the slowness, did they try to only use certain (base) plugins?

[guibou]

HLS is unusably slow for most of our developers, even those with 32GB of RAM.

I think some plugins contribute to the slowness, did they try to only use certain (base) plugins?

How do I disable plugins ? I.e. starting with the minimum and progressively increasing them? Is there a runtime option or should I just rebuild HLS without them?

[fendor]

I am not sure whether there exists a run-time option (there definitely should be), I built from source using cabal flags, and modified the constraints in cabal.project

[jneira]

The Haskell codebase at Mercury is 232kloc in a single cabal package with 1,600+ modules currently

in how many components (libs, test suites, benchmarks, execs) are distributed those modules?

[jneira]

To disable plugins at runtime you could use the lsp client provided json config, i will try to post a config example which disables all plugins.

[pepeiborra]

The Haskell codebase at Mercury is 232kloc in a single cabal package with 1,600+ modules currently. Build system is cabal with nix providing all dependencies.

A large number of packages probably contributes to slowdown, but even with a single package target, HLS is unusably slow for most of our developers, even those with 32GB of RAM.

Try this branch - #2060