Support IDE imports for multiple scripts/projects sharing the same project root

[jackkoenig]

Is your feature request related to a problem? Please describe.

It's common practice to have grab bags of "utility" scripts that are independent of each other. Often these are written in scripting languages like Python, Perl, or even Bash. Sometimes there will be shared code, sort of "library" files shared by multiple script "application" files. I am working on migrating such "grab bag utility" scripts to Scala Cli, it's awesome but this directory structure conflicts with Scala Cli's model of projects on a directory basis.

Note that it seems to work, except I think when I run different "scripts" in the same directory via scala-cli they are clobbering the .scala-build directory. The problem is most apparent when trying to use Metals with these scripts because it will report errors about multiple definitions (e.g. each top-level "script" has an object Main) and warnings about Using directives detected in multiple files. again due to multiple top-level "scripts".

Describe the solution you'd like

It would be nice if Scala Cli could treat projects on a per-file basis rather than per directory. using file directives in the top-file should be enough to define which files are in a particular project, and then Scala Cli could disambiguate each project based on the "top" file.

I am not sure if this is the same issue or separate, but the complaints about "using directives detected in multiple files" feels related--if I have some "library" files shared between separate projects, those libraries should be able to define their own dependencies. Maybe this is a step too much toward "build tool" land though.

Describe alternatives you've considered

I could define each "script" in its own directory, it makes replicating the behavior of grab bag "script" directories marginally more annoying, but I can just have each "script" actually be a tiny bash script that invokes the actual Scala code which is within its own directory. I already have to do this for various reasons so it's not that bad.

Additional context
Add any other context or screenshots about the feature request here.

[Gedochao]

@jackkoenig Do I understand correctly, that the issue is actually not about supporting file-based project structures (which to my knowledge is in place), but rather proper IDE imports for multiple scripts sharing the same project root?

Let me explain why do I think that.

It would be nice if Scala Cli could treat projects on a per-file basis rather than per directory.

I can see what you mean in this issue, but technically this is already the case. We do, in fact, treat projects on a per-file basis, rather than per-directory (unlike, for example, SBT)

Note that it seems to work, except I think when I run different "scripts" in the same directory via scala-cli they are clobbering the .scala-build directory.

Now, that may be so, but whatever lands in the .scala-build directory has separate Bloop project names. Incremental compilation problems should not occur and running different scripts in the same directory should not clash. If that would be the case, please raise it separately as a bug.

The problem is most apparent when trying to use Metals with these scripts because it will report errors about multiple definitions (e.g. each top-level "script" has an object Main) and warnings about Using directives detected in multiple files. again due to multiple top-level "scripts".

This is a problem completely separate from the .scala-build directory, and it's tied to the .bsp directory instead.
The .bsp directory contains the definition for importing the project to your IDE via the Build Server Protocol. The default location for that is in the root directory of the project (even if the project is, in fact, a single file).

Check the protocol definition:
https://build-server-protocol.github.io/docs/overview/server-discovery#default-locations-for-bsp-connection-files

Note that if multiple BSP connection files are found at .bsp/*.json, it is down to the BSP client (the IDE) how to handle the ambiguities (I do not in fact know of any BSP client which would handle that out of the box).
As things are right now, Scala CLI does not bother with generating a separate BSP connection if multiple projects (scripts) share a workspace. I'm not sure what would be the best way to handle that.

Theoretically, if the IDE was to support a separate classpath and configuration for each script in the directory, each of them would have to be imported as a separate connection. And then, if there would be source files/scripts reused in multiple projects living in the same directory (as described in the issue), then it likely would be unclear which configuration should be used. So generally speaking, it's easier said than done.

Also note that technically you could work around scripts sharing a workspace by making them have separate workspaces (even when they're in the same root directory). You can override where .bsp lands with the --bsp-dir command line option, or even the whole workspace (with .scala-build) via the --workspace command line option. I am aware it is not the user experience you'd have in mind, but I thought that could still be relevant.
It's also worth to keep in mind that for the IDE to pick up the right .bsp connection, it has to open the directory containing it (so all this workaround would do would be to separate those for scripts in a single directory - likely not what you had in mind).

I am not sure if this is the same issue or separate, but the complaints about "using directives detected in multiple files" feels related--if I have some "library" files shared between separate projects, those libraries should be able to define their own dependencies. Maybe this is a step too much toward "build tool" land though.

You can get rid of those with --suppress-directives-in-multiple-files-warning, or even suppress them permanently on your system with scala-cli config suppress-warning.directives-in-multiple-files true. The command line option could be integrated into your scripts' shebang. That being said, I agree we could perhaps tweak how this warning is being raised for scripts. Feel free to raise it as a separate issue.

Summoning @tgodzik and @kasiaMarek for the Metals perspective on the issue.

Generally speaking, this is something that was discussed here and there for quite some time, but as of now, no solve-it-all solution has been proposed.

[kasiaMarek]

I don't think there is anything scala-cli can do right now for Metals to go about thins the way you want it, unless it started supporting multiple build targets (which is very build tooly). Metals already treats a file without a build definition as single file scala-cli script. So this might be just about suppressing the scala-cli build... Anyway that sounds like a Metals feature request.

[tgodzik]

I was thinking about:

The problem is most apparent when trying to use Metals with these scripts because it will report errors about multiple definitions (e.g. each top-level "script" has an object Main) 

could we improve that though? Maybe have everything in a separate package for a script? That would at least make those scripts work.

Also if we do scala-cli run my-script.sc maybe we should not change .scala-build if the script is already included in the current paths?

So the workflow for Metals would be to include all files allow them to work within single module by default.

And for sure I would allow scripts to have using directives separate, so maybe not report it there.

On the Metals side we would have to start multiple server, which would mean more possible bugs and problems with that.

[Gedochao]

could we improve that though? Maybe have everything in a separate package for a script? That would at least make those scripts work.

@tgodzik what do you mean by a separate package?

Also if we do scala-cli run my-script.sc maybe we should not change .scala-build if the script is already included in the current paths?

.scala-build has to be changed, as that's where Bloop stores compilation data, among other things.

So the workflow for Metals would be to include all files allow them to work within single module by default.

Mixed feelings about this. Each of the scripts may have a very different classpath/configuration. It may improve the feeling overall, but it may also break existing scripts based on what directory they are in.

And for sure I would allow scripts to have using directives separate, so maybe not report it there.

Separate issue.

On the Metals side we would have to start multiple server, which would mean more possible bugs and problems with that.

I'm not sure if we can avoid that, however, if we want a full solution...