Fails to load with local tarballs in cabal.project

[tfausak]

Your environment

Which OS do you use?
Debian 11.4

Which version of GHC do you use and how did you install it?
GHC 9.0.2 from GHCup 0.1.18.0

How is your project built (alternative: link to the project)?
With Cabal

Which LSP client (editor/plugin) do you use?
VSCode 1.69.2 with extension haskell.haskell 2.2.0

Which version of HLS do you use and how did you install it?
Version 1.7.0.0 from GHCup

Have you configured HLS in any way (especially: a hie.yaml file)?
Simple Cabal cradle in hie.yaml, but the problem happens even without it

Steps to reproduce

Put a local tarball as a package in the cabal.project file. For example:

-- cabal.project
packages:
  .
  vendor/flow-2.0.0.0.tar.gz

Expected behaviour

The project should load, including the vendored tarball as a dependency.

Actual behaviour

The project fails to load.

Debug information

From the command line:

haskell-language-server-wrapper: .../vendor/vendor/flow-2.0.0.0.tar.gz: getDirectoryContents:openDirStream: inappropriate type (Not a directory)

From the extension:

[Error - 12:36:46 PM] setInitialDynFlags: .../vendor/flow-2.0.0.0.tar.gz: getDirectoryContents:openDirStream: inappropriate type (Not a directory)

Link to documentation about tarballs in cabal.project: https://cabal.readthedocs.io/en/3.6/cabal-project.html#cfg-field-packages

[tfausak]

Oh, a few other things about this:

• HLS appears to work anyway, in spite of the problem.
• Using file://... syntax does not have the same problem, but you have to specify absolute paths.
• Using an http://... source does not have the same problem.

[codygman]

This is because of gen-hie you can get from installing from cabal install implicit-hie. Should probably also open a bug there or at least ping @Avi-D-coder.

[codygman]

This issue can be avoided if you already have a working hie.yaml because gen-hie will be run and display an error, but hls will carry on anyway apparently (as it should since it handles tarballs fine).

A workaround then is:

• temporarily remove all tarball lines like vendor/flow-2.0.0.0.tar.gz
• Install implicit-hie and run gen-hie
• add back your tarball lines
• hls should run fine

[fendor]

I don't recommend running gen-hie for cabal projects.
Just use the simple cabal cradle, e.g.

cradle:
  cabal:

I believe that should work for any cabal based project.

See #2342 where implicit-hie delegates for cabal projects to this cradle type.

[codygman]

I don't recommend running gen-hie for cabal projects. Just use the simple cabal cradle, e.g.

cradle:
  cabal:

I believe that should work for any cabal based project.

See #2342 where implicit-hie delegates for cabal projects to this cradle type.

I'm pretty sure I tested that one first and It won't work if you have a tarball as described.

I'll recheck.

[fendor]

@codygman any updates? I can not imagine that the simple cradle fails

[codygman]

@fendor I've been using it with the simple cradle today for most of the day.

Observations:

• I still get the tarball error from this issue
• hls crashes more often (maybe my none-cradles need to be added to the simple cabal cradle?)

[fendor]

Can you share a project or something? I am curious why it doesnt work, there is nothing about this that should affect HLS.

[codygman]

Nothing special about the project, however I wasn't able to reproduce just now for whatever reason with flow:

shell session creating cabal project/using local tarball

$ cabal get flow && cd flow-2.0.0.0/ && cabal sdist
Unpacking to flow-2.0.0.0/

Wrote tarball sdist to
/tmp/flow-2.0.0.0/dist-newstyle/sdist/flow-2.0.0.0.tar.gz

flow-2.0.0.0]$ mkdir /tmp/new-hs-proj && cd /tmp/new-hs-proj/ && cabal init

Guessing dependencies...

Generating LICENSE...
Warning: unknown license type, you must put a copy in LICENSE yourself.
Generating CHANGELOG.md...
Generating app/Main.hs...
Generating new-hs-proj.cabal...

Warning: no synopsis given. You should edit the .cabal file and add one.
You may want to edit the .cabal file and add a Description field.

$ mv /tmp/flow-2.0.0.0/dist-newstyle/sdist/flow-2.0.0.0.tar.gz .

$ cat cabal.project 
packages:
  .
  flow-2.0.0.0.tar.gz
$ cabal run
# ... snip ...
$ haskell-language-server --probe-tools && cat /etc/issue
2022-08-14T05:33:25.754358Z | Info | No log file specified; using stderr.
haskell-language-server version: 1.7.0.0 (GHC: 9.0.2) (PATH: /nix/store/1k7vwd254mp20lyp9c05my7czly54z1n-haskell-language-server-1.7.0.0/bin/haskell-language-server)
Tool versions found on the $PATH
cabal:		3.6.2.0
stack:		Not found
ghc:		9.0.2

<<< Welcome to NixOS 22.11.20220810.a7f89dd (\m) - \l >>>

$ haskell-language-server
2022-08-14T05:33:30.968996Z | Info | No log file specified; using stderr.
2022-08-14T05:33:30.969435Z | Info | haskell-language-server version: 1.7.0.0 (GHC: 9.0.2) (PATH: /nix/store/1k7vwd254mp20lyp9c05my7czly54z1n-haskell-language-server-1.7.0.0/bin/haskell-language-server)
2022-08-14T05:33:30.970100Z | Info | Directory: /tmp/new-hs-proj
2022-08-14T05:33:30.970329Z | Info | Logging heap statistics every 60.00s
 ghcide setup tester in /tmp/new-hs-proj.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /tmp/new-hs-proj
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  ()

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2022-08-14T05:33:31.071963Z | Info | Cradle path: app/Main.hs
2022-08-14T05:33:31.072074Z | Warning | No [cradle](https://github.com/mpickering/hie-bios#hie-bios) found for app/Main.hs.
Proceeding with [implicit cradle](https://hackage.haskell.org/package/implicit-hie).
You should ignore this message, unless you see a 'Multi Cradle: No prefixes matched' error.
2022-08-14 00:33:32.202990097 [ThreadId 34] INFO hie-bios:	Resolving dependencies...
2022-08-14 00:33:32.418098798 [ThreadId 34] INFO hie-bios:	Build profile: -w ghc-9.0.2 -O1
2022-08-14 00:33:32.418178714 [ThreadId 34] INFO hie-bios:	In order, the following will be built (use -v for more details):
2022-08-14 00:33:32.418234777 [ThreadId 34] INFO hie-bios:	 - new-hs-proj-0.1.0.0 (exe:new-hs-proj) (first run)
2022-08-14 00:33:32.44049266 [ThreadId 34] INFO hie-bios:	Configuring executable 'new-hs-proj' for new-hs-proj-0.1.0.0..
2022-08-14 00:33:33.18775907 [ThreadId 34] INFO hie-bios:	Preprocessing executable 'new-hs-proj' for new-hs-proj-0.1.0.0..
2022-08-14T05:33:33.547228Z | Info | Interface files cache directory: /home/cody/.cache/ghcide/main-d140beff4fdd743ca2bb1fe812875474b153930a
2022-08-14T05:33:33.547566Z | Info | Making new HscEnv. In-place unit ids: [main]

Completed (1 file worked, 0 files failed)

Let me try amazonka. @tfausak I'm assuming you generated the tarball using cabal sdist?

[codygman]

It also worked for amazonka with the below script:

cd /tmp && cabal get amazonka-core && cd amazonka-core-1.6.1 && cabal sdist
cd /tmp && cabal get amazonka-sns && cd amazonka-sns-1.6.1 && cabal sdist
mkdir /tmp/new-hs-proj-az && cd /tmp/new-hs-proj-az/ && cabal init
mv /tmp/amazonka-core-1.6.1/dist-newstyle/sdist/amazonka-core-1.6.1.tar.gz .
mv /tmp/amazonka-sns-1.6.1/dist-newstyle/sdist/amazonka-sns-1.6.1.tar.gz .
cat cabal.project
cabal run

script results

[nix-shell:/tmp]$ cd /tmp && cabal get amazonka-core && cd amazonka-core-1.6.1 && cabal sdist
Unpacking to amazonka-core-1.6.1/
Wrote tarball sdist to
/tmp/amazonka-core-1.6.1/dist-newstyle/sdist/amazonka-core-1.6.1.tar.gz

[nix-shell:/tmp/amazonka-core-1.6.1]$ cd /tmp && cabal get amazonka-sns && cd amazonka-sns-1.6.1 && cabal sdist
Unpacking to amazonka-sns-1.6.1/
Wrote tarball sdist to
/tmp/amazonka-sns-1.6.1/dist-newstyle/sdist/amazonka-sns-1.6.1.tar.gz

[nix-shell:/tmp/amazonka-sns-1.6.1]$ cd /tmp/new-hs-proj-az/

[nix-shell:/tmp/new-hs-proj-az]$ mv /tmp/amazonka-sns-1.6.1/dist-newstyle/sdist/amazonka-sns-1.6.1.tar.gz .

[nix-shell:/tmp/new-hs-proj-az]$ cat cabal.project
packages:
  .
  amazonka-core-1.6.1.tar.gz
  amazonka-sns-1.6.1.tar.gz
[nix-shell:/tmp/new-hs-proj-az]$ haskell-language-server
2022-08-14T05:49:23.962743Z | Info | No log file specified; using stderr.
2022-08-14T05:49:23.963058Z | Info | haskell-language-server version: 1.7.0.0 (GHC: 9.0.2) (PATH: /nix/store/1k7vwd254mp20lyp9c05my7czly54z1n-haskell-language-server-1.7.0.0/bin/haskell-language-server)
2022-08-14T05:49:23.963728Z | Info | Directory: /tmp/new-hs-proj-az
2022-08-14T05:49:23.963959Z | Info | Logging heap statistics every 60.00s
 ghcide setup tester in /tmp/new-hs-proj-az.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /tmp/new-hs-proj-az
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  (/tmp/new-hs-proj-az/hie.yaml)

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2022-08-14T05:49:23.968400Z | Info | Cradle path: app/Main.hs
2022-08-14 00:49:26.905156228 [ThreadId 34] INFO hie-bios:	Resolving dependencies...
2022-08-14 00:49:27.980219501 [ThreadId 34] INFO hie-bios:	Build profile: -w ghc-9.0.2 -O1
2022-08-14 00:49:27.98031712 [ThreadId 34] INFO hie-bios:	In order, the following will be built (use -v for more details):
2022-08-14 00:49:27.980479483 [ThreadId 34] INFO hie-bios:	 - new-hs-proj-az-0.1.0.0 (exe:new-hs-proj-az) (first run)
2022-08-14 00:49:28.013248803 [ThreadId 34] INFO hie-bios:	Preprocessing executable 'new-hs-proj-az' for new-hs-proj-az-0.1.0.0..
2022-08-14T05:49:28.665821Z | Info | Interface files cache directory: /home/cody/.cache/ghcide/main-3d0a5703c74d48a4a2c919e0b28b2d266fd38127
2022-08-14T05:49:28.666127Z | Info | Making new HscEnv. In-place unit ids: [main]

Completed (1 file worked, 0 files failed)

[nix-shell:/tmp/new-hs-proj-az]$ haskell-language-server --probe-tools && cat /etc/issue
2022-08-14T05:51:28.902059Z | Info | No log file specified; using stderr.
haskell-language-server version: 1.7.0.0 (GHC: 9.0.2) (PATH: /nix/store/1k7vwd254mp20lyp9c05my7czly54z1n-haskell-language-server-1.7.0.0/bin/haskell-language-server)
Tool versions found on the $PATH
cabal:		3.6.2.0
stack:		Not found
ghc:		9.0.2


<<< Welcome to NixOS 22.11.20220810.a7f89dd (\m) - \l >>>

Run 'nixos-help' for the NixOS manual.

Version/system info

haskell-language-server --probe-tools && cat /etc/issue
haskell-language-server

[tfausak]

I think this is a bug with the VSCode extension, not HLS. I made a Gist to reproduce this problem: https://gist.github.com/tfausak/381ccd98ef008ecbe975777f6ea9429b

Running HLS works fine, but VSCode shows the error and continues to work in spite of that.

[codygman]

I can confirm I get the same in emacs lsp events buffer, but not using hls on CLI.

So it's an lsp issue?

[codygman]

This issue still happens in our company codebase.

I tried making a small reproduction of it here:

https://github.com/codygman/repro-local-tarball-failure-hls-3074

However for whatever reason... it's not reproducing. Maybe I should enable template haskell and other extensions.

[codygman]

Actually, I've updated a template that closely resembles our company codebase and cannot reproduce the issue in it's devcontainer here:

https://github.com/acilearning/aci-docker-haskell-template

What about you @tfausak?

Though I do see these errors:

2023-02-17T21:56:43.008013Z | Warning | explicit-fields: InternalError: Unable to TypeCheck
[Warn  - 9:56:43 PM] explicit-fields: InternalError: Unable to TypeCheck

[tfausak]

I posted a minimal example here: #3074 (comment)

I just updated it to use newer versions of everything: https://gist.github.com/tfausak/381ccd98ef008ecbe975777f6ea9429b/63015d4dabac80891d895f0f028bc135247228a5

I can still reproduce the problem:

I'm using:

• macOS 13.2.1
• Docker Desktop 4.16.2
• VSCode 1.75.1
• Haskell extension 2.2.2
• GHCup 0.1.19.0
• GHC 9.4.4
• Cabal 3.8.1.0
• HLS 1.9.1.0

[codygman]

With your updated gist I can reproduce in both vscode and emacs but not when running hls from command line.

So I believe:

• It's not vscode extension specific because it happens on emacs too
• it is lsp specific because it doesn't happen from command line
• it doesn't happen from the command line with the --lsp option so some lsp request from editors triggers this

A log of the failure from emacs using eglot:

(:jsonrpc "2.0" :method "window/logMessage" :params
	  (:message "setInitialDynFlags: /workspaces/example/flow-2.0.0.2.tar.gz: getDirectoryContents:openDirStream: inappropriate type (Not a directory)" :type 1))

Showing that error doesn't happen from command line:

❯ docker compose run -v $(pwd):/workspace/example devcontainer bash -c "haskell-language-server-wrapper"
Found "/workspace/example/hie.yaml" for "/workspace/example/a"
Run entered for haskell-language-server-wrapper(haskell-language-server-wrapper) Version 1.9.1.0 aarch64 ghc-9.4.4
Current directory: /workspace/example
Operating system: linux
Arguments: []
Cradle directory: /workspace/example
Cradle type: Cabal

Tool versions found on the $PATH
cabal:          3.8.1.0
stack:          2.9.3
ghc:            9.4.4


Consulting the cradle to get project GHC version...
Project GHC version: 9.4.4
haskell-language-server exe candidates: ["haskell-language-server-9.4.4","haskell-language-server"]
Launching haskell-language-server exe at:/home/haskell/.ghcup/bin/haskell-language-server-9.4.4
2023-02-20T16:20:02.950016Z | Info | No log file specified; using stderr.
2023-02-20T16:20:02.951164Z | Info | haskell-language-server version: 1.9.1.0 (GHC: 9.4.4) (PATH: /home/haskell/.ghcup/hls/1.9.1.0/lib/haskell-language-server-1.9.1.0/bin/haskell-language-server-9.4.4)
2023-02-20T16:20:02.952228Z | Info | Directory: /workspace/example
2023-02-20T16:20:02.952491Z | Info | Logging heap statistics every 60.00s
 ghcide setup tester in /workspace/example.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Step 1/4: Finding files to test in /workspace/example
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle
  (/workspace/example/hie.yaml)

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
2023-02-20T16:20:02.976454Z | Info | Cradle path: Example.hs
2023-02-20T16:20:04.467553Z | Info | Interface files cache directory: /home/haskell/.cache/ghcide/example-0-inplace-2e514414be0eb14616a18a9c3fd0ae0a232176bc
2023-02-20T16:20:04.468234Z | Info | Making new HscEnv. In-place unit ids: [example-0-inplace]

Completed (1 file worked, 0 files failed)

[codygman]

I searched setInitialDynFlags and found use implicit-hie cradle from setInitialDynFlags from #1375.

I suppose that's why the issue from the implicit-hie maintainer was created in Avi-D-coder/implicit-hie#47?

So I guess they believe or know it to be an implicit-hie issue?

They probably know better than I do, but the place I could find I think is most relevant to this issue is right before the problematic setInitialDynFlags log is here.

[Avi-D-coder]

@codygman I unfortunately don't have the time to look into this issue, contributions to implicit-hie are always welcome.
However I'm not sure this issue is in implicit-hie, the null cradle should use cabals built in info not implicit-hie, so if your getting this error with a null cradle, and whatever implicit-hie gen-hie > hie.yaml produces then the issue is likely not in implicit-hie. setInitialDynFlags looks like the right place to start looking.

[wz1000]

It seems like setInitialDynFlags calls findCradle from hie-bios with the wrong kind of argument. findCradle expects a haskell source file, but we give it the root directory. This means that hie-bios then looks for the hie.yaml file in the parent of the root directory.
Due to this, we fail to find the appropriate cradle, and default to an implicit cradle from implicit-hie, where the bug is probably triggered.

The following patch seems to make the reproducer work in HLS:

diff --git a/ghcide/session-loader/Development/IDE/Session.hs b/ghcide/session-loader/Development/IDE/Session.hs
index 2f2f9fcf3..155de1f6a 100644
--- a/ghcide/session-loader/Development/IDE/Session.hs
+++ b/ghcide/session-loader/Development/IDE/Session.hs
@@ -260,7 +260,7 @@ loadWithImplicitCradle mHieYaml rootDir = do
 getInitialGhcLibDirDefault :: Recorder (WithPriority Log) -> FilePath -> IO (Maybe LibDir)
 getInitialGhcLibDirDefault recorder rootDir = do
   let log = logWith recorder
-  hieYaml <- findCradle def rootDir
+  hieYaml <- findCradle def (rootDir </> "a.hs")
   cradle <- loadCradle def hieYaml rootDir
   libDirRes <- getRuntimeGhcLibDir cradle
   case libDirRes of

Obviously this is not a patch we can merge into HLS, so I think findCradle should be changed to test if its argument is a directory or a file, and only start searching in the parents if it is a file. If the argument is a directory, it should start searching from the hie.yaml in the directory itself, not its parents.

implicit-hie also deserves finer examination to diagnose the root cause of the exception.

/cc @fendor

[fendor]

Obviously this is not a patch we can merge into HLS, so I think findCradle should be changed to test if its argument is a directory or a file, and only start searching in the parents if it is a file.

In the wrapper, we already add a magic filepath https://github.com/haskell/haskell-language-server/blob/master/exe/Wrapper.hs#L222

In any way, I don't recall whether there was a reason findCradle (other than API consistency) couldnt check whether its parameter is a directory. I can implement the proposed change and see what the test-suite says.