HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found #3822

MorrowM · 2023-10-01T18:41:11Z

I've had two people in the FP Discord haskell-tooling channel having issues with HLS 2.3.0.0 on Windows. After some troubleshooting we see there's an issue involving libstdc++-6.dll not being found.

Your environment

Which OS do you use?
Windows 11
Which version of GHC do you use and how did you install it?
9.4.7
How is your project built (alternative: link to the project)?
(Not relevant)
Which LSP client (editor/plugin) do you use?
VSCode+vscode-haskell
Which version of HLS do you use and how did you install it?
2.3.0.0 with ghcup install hls 2.3.0.0
Have you configured HLS in any way (especially: a hie.yaml file)?
No

Steps to reproduce

Simply run haskell-language-server-wrapper-2.3.0.0.exe in a cmd window (powershell crashes silently, cmd shows an actual error).

Expected behaviour

Debug output is displayed.

Actual behaviour

We get the following error window popping up:

Additional information

This manifests in VSCode by displaying "The Haskell (PF) server crashed 5 times in the last 3 minutes. The server will not be restarted." The output pane shows:

(Sorry for the screenshot, this is what I have from the conversation.)

I suspect #3820 might be related (possibly they had 2.2.0.0 set, which is why running haskell-language-server-wrapper didn't reveal the issue).

The text was updated successfully, but these errors were encountered:

Bodigrim · 2023-10-01T19:29:51Z

(Just passing by)
Check which hls dependencies involve C++, which manifests in Cabal files as extra-libraries: stdc++. A proper way to depend on it is fairly arcane:

    if impl(ghc >= 9.4)
      build-depends: system-cxx-std-lib == 1.0
    elif os(darwin) || os(freebsd)
      extra-libraries: c++
    elif os(openbsd)
      extra-libraries: c++ c++abi pthread
    elif os(windows)
      -- GHC's Windows toolchain is based on clang/libc++ in GHC 9.4 and later
      if impl(ghc < 9.3)
        extra-libraries: stdc++
      else
        extra-libraries: c++ c++abi
    elif arch(wasm32)
      cxx-options: -fno-exceptions
      extra-libraries: c++ c++abi
    else
      extra-libraries: stdc++

If a package violates this (e. g., declares extra-libraries: stdc++ unconditionally), the bindist is likely to be broken, especially on Windows.

michaelpj · 2023-10-01T21:05:58Z

cc @wz1000

KripkesBeard · 2023-10-02T01:55:33Z

I have the exact same issue (Windows 10 but otherwise same environment) and have the same output logs, with the exception that haskell-language-server-wrapper-2.3.0.0.exe in cmd crashes with the text "The application was unable to start correctly (0xc000007b). Click OK to close the application." as an application error, not a systems error.

I had hls 2.2.0.0 set as default without 2.3.0.0 installed, and when I opened a Haskell project in VSCode it prompted me to install 2.3.0.0, which I did, and after installing it has been crashing on start up with the same output as the screenshot above.

Edit:

I uninstalled 2.3.0.0 and tried to reproduce the error to see what the HLS logs are if I said no to installing 2.3.0.0 and this is what I got

[client] INFO Finding haskell-language-server
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose upgrade' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t hls -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t cabal -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t stack -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis hls 2.2.0.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis cabal 3.8.1.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis stack 2.9.1' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Executing 'ghc.exe --numeric-version' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose run --hls 2.2.0.0 --cabal 3.8.1.0 --stack 2.9.1 --install' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Working out the project GHC version. This might take a while...
[client] INFO Executing 'haskell-language-server-wrapper --project-ghc-version' in cwd 'd:\GitHub\fragments'
[client] INFO The GHC version for the project or file: 9.4.7
[client] INFO Platform constants: Windows, A_64
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t hls -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis bindir' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis hls 2.3.0.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] ERROR Error executing 'ghcup --no-verbose whereis hls 2.3.0.0' with error code 30
[client] ERROR stderr: esc[1m[ Error ]esc[0m [sec]8;;https://errors.haskell.org/messages/GHCup-00130esc\GHCup-00130esc]8;;sec] The version 2.3.0.0 of the tool hls is not installed.

which prompts me to install 2.3.0.0 and when I say no

[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis ghc 9.4.7' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] ERROR Error: Project requires HLS but it isn't installed
at c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:61763
at Array.forEach (<anonymous>)
at c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:61709
at Generator.next (<anonymous>)
at s (c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:49145)

angerman · 2023-10-02T03:14:51Z

Even

-- GHC's Windows toolchain is based on clang/libc++ in GHC 9.4 and later

from @Bodigrim's comment is not necessarily true. The toolchain GHC bindists for windows ship, ship by default for UCRT, not MSVCRT from 9.4 onwards anymore. However this does not mean GHC 9.4+ can't be built against MSVCRT anymore. The whole assumption that GHC and the toolchain are bundled is deeply flawed imo. Yes, we do ship a specific toolchain with GHC in our windows binary distributions, but making the universal assumption that everyone depends on this is some tight coupling I'd rather we don't do.

Also you can use UCRT but with the Gnu toolchain. MSYS2 classifies those environments as UCRT64 (ucr64 + ghc) and CLANG64 (ucrt64 + llvm).

There is system-cxx-std-lib in recent GHCs trying to work around this to some degree.

andreabedini · 2023-10-02T06:31:06Z

I tried to find the misbeaving package that imports stdc++ unconditionally, as @Bodigrim suggests.

I downloaded the install plans for the release from https://github.com/haskell/haskell-language-server/releases/download/2.3.0.0/plan_json.tar and used a simple script to download all the cabal files involved.

$ mkdir x86_64-mingw64-ghc-9.4.7-plan; cat x86_64-mingw64-ghc-9.4.7-plan.json | jq -r '."install-plan"[] | select(."pkg-cabal-sha256") | [.id, ."pkg-cabal-sha256"] | @tsv' | parallel --bar --colsep '\t' 'curl --silent -o x86_64-mingw64-ghc-9.4.7-plan/{1}.cabal https://casa.stackage.org/{2}'

... but grepping for stdc++ does not return anything 🤔

PS: perhaps we should upload the build-infos too!

angerman · 2023-10-02T12:38:21Z

we could just ship the libstdc++6.dll alongside.

GuenterHummel · 2023-10-02T14:16:00Z

libgcc_s_seh-1. dll and libwinpthread-1.dll are also missing, I think. it was linked dynamically and not statically as usual

Bodigrim · 2023-10-02T22:19:20Z

@andreabedini text does depend on stdc++, so at least one package should have been found by grep.

Which binaries are affected? This can be checked with ldd.

wz1000 · 2023-10-03T09:48:32Z

Here's a summary of the problem

HLS 2.3 is the first version to ship with text-2.0 for GHC 9.2 and 9.0
We link system libraries dynamically on windows
It is currently completely non-deterministic which version of hls-wrapper we ship (i.e. whichever job finishes the last, we ship that binary)

On GHC 9.0 and 9.2, text links against stdc++ directly, on 9.4+ it depends on system-cxx-lib

As a result, the haskell-language-server-9.0.2, haskell-language-server-9.2.8 and haskell-language-server-wrapper executables are broken.

Possible sollutions are:

Link system libraries statically on windows, like we already do Haskell libraries (by setting static: True in the cabal.project
avoid c++ dependency by using text -simdutf
Ship stdc++.dll along with the binaries, but this might be complicated because we might want to ship different versions for each executable depending on what toolchain/ghc version they support

1 or 2 seem the easiest to do. We should also modify CI to ensure that we can control which version of haskell-language-server-wrapper (i.e. which GHC it is built with) we ship.

andreabedini · 2023-10-05T03:43:02Z

@andreabedini text does depend on stdc++, so at least one package should have been found by grep.

@Bodigrim Good point. It turns out that plan uses the pre-installed text, so text's cabal file is not specified in the plan.

On GHC 9.0 and 9.2, text links against stdc++ directly, on 9.4+ it depends on system-cxx-lib

Doesn't what I wrote above plus this imply that the problem could be with the ghc bindist?
We have evicence that hls-2.3.0.0 for x86_64-mingw64-ghc-9.4.7 was compiled using the pre-installed text library from the GHC binary distribution:

❯ cabal-plan-filter.jq pkg-name:text < x86_64-mingw64-ghc-9.4.7-plan.json
{
  "type": "pre-existing",
  "id": "text-2.0.2",
  "pkg-name": "text",
  "pkg-version": "2.0.2",
  "depends": [
    "array-0.5.4.0",
    "base-4.17.2.0",
    "binary-0.8.9.1",
    "bytestring-0.11.5.2",
    "deepseq-1.4.8.0",
    "ghc-prim-0.9.1",
    "template-haskell-2.19.0.0"
  ]
}

And, if I am not mistaken, text depends on system-cxx-std-lib on ghc-9.4

    if impl(ghc >= 9.4)
      build-depends: system-cxx-std-lib == 1.0

Therefore stdc++ should be part of the GHC bindist?

I am confused.

wz1000 · 2023-10-05T05:40:14Z

@andreabedini the haskell-language-server-9.4.7 binary seems to work for me, the 9.2 and 9.0 binaries are the ones that seem broken. Is this not the case for you?

andreabedini · 2023-10-05T09:10:11Z

@wz1000 I am on linux :) I was only trying to troubleshoot. I picked x86_64-mingw64-ghc-9.4.7 from the information in the OP.

bflyblue · 2023-10-05T09:21:57Z

Happy to help with debugging this on windows.

c:\ghcup>msys64\usr\bin\ldd.exe "bin\haskell-language-server-9.0.2~2.3.0.0.exe"
        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x7ff8626f0000)
        KERNEL32.DLL => /c/Windows/System32/KERNEL32.DLL (0x7ff861980000)
        KERNELBASE.dll => /c/Windows/System32/KERNELBASE.dll (0x7ff85fe60000)
        apphelp.dll => /c/Windows/SYSTEM32/apphelp.dll (0x7ff85ce20000)
        ADVAPI32.dll => /c/Windows/System32/ADVAPI32.dll (0x7ff860450000)
        msvcrt.dll => /c/Windows/System32/msvcrt.dll (0x7ff860fa0000)
        sechost.dll => /c/Windows/System32/sechost.dll (0x7ff861070000)
        RPCRT4.dll => /c/Windows/System32/RPCRT4.dll (0x7ff861120000)
        GDI32.dll => /c/Windows/System32/GDI32.dll (0x7ff861aa0000)
        dbghelp.dll => /c/Windows/SYSTEM32/dbghelp.dll (0x7ff851980000)
        win32u.dll => /c/Windows/System32/win32u.dll (0x7ff85fd70000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff860330000)
        gdi32full.dll => /c/Windows/System32/gdi32full.dll (0x7ff860210000)
        combase.dll => /c/Windows/System32/combase.dll (0x7ff8612f0000)
        msvcp_win.dll => /c/Windows/System32/msvcp_win.dll (0x7ff85fbe0000)
        OLEAUT32.dll => /c/Windows/System32/OLEAUT32.dll (0x7ff8620a0000)
        USER32.dll => /c/Windows/System32/USER32.dll (0x7ff861680000)
        ole32.dll => /c/Windows/System32/ole32.dll (0x7ff861e80000)
        SHELL32.dll => /c/Windows/System32/SHELL32.dll (0x7ff860550000)
        SHLWAPI.dll => /c/Windows/System32/SHLWAPI.dll (0x7ff861e20000)
        WS2_32.dll => /c/Windows/System32/WS2_32.dll (0x7ff862020000)
        libstdc++-6.dll => not found
        WINMM.dll => /c/Windows/SYSTEM32/WINMM.dll (?)

c:\ghcup>msys64\usr\bin\ldd.exe "bin\haskell-language-server-9.4.7~2.3.0.0.exe"
        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x7ff8626f0000)
        KERNEL32.DLL => /c/Windows/System32/KERNEL32.DLL (0x7ff861980000)
        KERNELBASE.dll => /c/Windows/System32/KERNELBASE.dll (0x7ff85fe60000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff860330000)
        SHELL32.dll => /c/Windows/System32/SHELL32.dll (0x7ff860550000)
        msvcp_win.dll => /c/Windows/System32/msvcp_win.dll (0x7ff85fbe0000)
        USER32.dll => /c/Windows/System32/USER32.dll (0x7ff861680000)
        win32u.dll => /c/Windows/System32/win32u.dll (0x7ff85fd70000)
        GDI32.dll => /c/Windows/System32/GDI32.dll (0x7ff861aa0000)
        gdi32full.dll => /c/Windows/System32/gdi32full.dll (0x7ff860210000)
        SHLWAPI.dll => /c/Windows/System32/SHLWAPI.dll (0x7ff861e20000)
        msvcrt.dll => /c/Windows/System32/msvcrt.dll (0x7ff860fa0000)
        ADVAPI32.dll => /c/Windows/System32/ADVAPI32.dll (0x7ff860450000)
        sechost.dll => /c/Windows/System32/sechost.dll (0x7ff861070000)
        dbghelp.dll => /c/Windows/SYSTEM32/dbghelp.dll (0x7ff851980000)
        RPCRT4.dll => /c/Windows/System32/RPCRT4.dll (0x7ff861120000)
        combase.dll => /c/Windows/System32/combase.dll (0x7ff8612f0000)
        WS2_32.dll => /c/Windows/System32/WS2_32.dll (0x7ff862020000)
        OLEAUT32.dll => /c/Windows/System32/OLEAUT32.dll (0x7ff8620a0000)
        ole32.dll => /c/Windows/System32/ole32.dll (0x7ff861e80000)
        WINMM.dll => /c/Windows/SYSTEM32/WINMM.dll (0x7ff858f50000)
        dbgcore.DLL => /c/Windows/SYSTEM32/dbgcore.DLL (0x7ff83d630000)

c:\ghcup>msys64\usr\bin\ldd.exe "bin\haskell-language-server-wrapper-2.3.0.0.exe"
        ntdll.dll => /c/Windows/SYSTEM32/ntdll.dll (0x7ff8626f0000)
        KERNEL32.DLL => /c/Windows/System32/KERNEL32.DLL (0x7ff861980000)
        KERNELBASE.dll => /c/Windows/System32/KERNELBASE.dll (0x7ff85fe60000)
        ADVAPI32.dll => /c/Windows/System32/ADVAPI32.dll (0x7ff860450000)
        msvcrt.dll => /c/Windows/System32/msvcrt.dll (0x7ff860fa0000)
        sechost.dll => /c/Windows/System32/sechost.dll (0x7ff861070000)
        RPCRT4.dll => /c/Windows/System32/RPCRT4.dll (0x7ff861120000)
        GDI32.dll => /c/Windows/System32/GDI32.dll (0x7ff861aa0000)
        win32u.dll => /c/Windows/System32/win32u.dll (0x7ff85fd70000)
        dbghelp.dll => /c/Windows/SYSTEM32/dbghelp.dll (0x7ff851980000)
        gdi32full.dll => /c/Windows/System32/gdi32full.dll (0x7ff860210000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff860330000)
        msvcp_win.dll => /c/Windows/System32/msvcp_win.dll (0x7ff85fbe0000)
        combase.dll => /c/Windows/System32/combase.dll (0x7ff8612f0000)
        USER32.dll => /c/Windows/System32/USER32.dll (0x7ff861680000)
        OLEAUT32.dll => /c/Windows/System32/OLEAUT32.dll (0x7ff8620a0000)
        ole32.dll => /c/Windows/System32/ole32.dll (0x7ff861e80000)
        SHELL32.dll => /c/Windows/System32/SHELL32.dll (0x7ff860550000)
        SHLWAPI.dll => /c/Windows/System32/SHLWAPI.dll (0x7ff861e20000)
        WS2_32.dll => /c/Windows/System32/WS2_32.dll (0x7ff862020000)
        libstdc++-6.dll => not found
        WINMM.dll => /c/Windows/SYSTEM32/WINMM.dll (?)

…, especially on older GHCs. See #3822.

wz1000 · 2023-10-11T06:10:47Z

Fixed by e79a030

MorrowM added status: needs triage type: bug Something isn't right: doesn't work as intended, documentation is missing/outdated, etc.. labels Oct 1, 2023

theGhostJW mentioned this issue Oct 3, 2023

HLS crashing on startup after update from: 2.2.0.0 => 2.3.0.0 (Windows) #3820

Closed

wz1000 added a commit that referenced this issue Oct 10, 2023

Disable simdutf for text globally. C++ is hard to distribute properly…

d010614

…, especially on older GHCs. See #3822.

This was referenced Oct 10, 2023

Release 2.4.0.0 #3834

Merged

HLS 2.4.0.0 haskell/ghcup-metadata#141

Merged

wz1000 added a commit that referenced this issue Oct 11, 2023

Disable simdutf for text globally. C++ is hard to distribute properly…

e79a030

…, especially on older GHCs. See #3822.

wz1000 closed this as completed Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found #3822

HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found #3822

MorrowM commented Oct 1, 2023

Bodigrim commented Oct 1, 2023

michaelpj commented Oct 1, 2023

KripkesBeard commented Oct 2, 2023 •

edited

Loading

angerman commented Oct 2, 2023 •

edited

Loading

andreabedini commented Oct 2, 2023

angerman commented Oct 2, 2023

GuenterHummel commented Oct 2, 2023

Bodigrim commented Oct 2, 2023

wz1000 commented Oct 3, 2023 •

edited

Loading

andreabedini commented Oct 5, 2023 •

edited

Loading

wz1000 commented Oct 5, 2023

andreabedini commented Oct 5, 2023

bflyblue commented Oct 5, 2023

wz1000 commented Oct 11, 2023

HLS 2.3.0.0 release possibly borked on Windows - libstdc++-6.dll not found #3822

HLS 2.3.0.0 release possibly borked on Windows - libstdc++-6.dll not found #3822

Comments

MorrowM commented Oct 1, 2023

Your environment

Steps to reproduce

Expected behaviour

Actual behaviour

Additional information

Bodigrim commented Oct 1, 2023

michaelpj commented Oct 1, 2023

KripkesBeard commented Oct 2, 2023 • edited Loading

angerman commented Oct 2, 2023 • edited Loading

andreabedini commented Oct 2, 2023

angerman commented Oct 2, 2023

GuenterHummel commented Oct 2, 2023

Bodigrim commented Oct 2, 2023

wz1000 commented Oct 3, 2023 • edited Loading

andreabedini commented Oct 5, 2023 • edited Loading

wz1000 commented Oct 5, 2023

andreabedini commented Oct 5, 2023

bflyblue commented Oct 5, 2023

wz1000 commented Oct 11, 2023

HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found #3822

HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found #3822

KripkesBeard commented Oct 2, 2023 •

edited

Loading

angerman commented Oct 2, 2023 •

edited

Loading

wz1000 commented Oct 3, 2023 •

edited

Loading

andreabedini commented Oct 5, 2023 •

edited

Loading