Extra semanticTokens requests in a multiroot workspace

[andyw8]

👋

I'm a maintainer on Ruby LSP.

I have a multiroot repo with two apps, app1 and app2. https://github.com/andyw8/multiroot_monorepo

While a file from app1 is open, the logs show all requests going to app1, as expected, except for one type of request, semanticTokens:

This happens even if I have no files from app2 open.

Is there something special about semanticTokens that might cause it to behave that way?

(I can provide further information to help troubleshoot, but wanted to ask this up-front in case there's a simple answer).

cc @vinistock

[vinistock]

Worth pointing out that the Ruby LSP spawns separate language servers for each configured workspace. Each one of the clients has their document selector configured to only handle requests for the specific workspace they are related to.

All other requests work as expected: each language server only receives requests for files that are inside their workspace. Except for semantic highlighting, which seems to be sent to all servers despite the document selector.

[dbaeumer]

Is this still an issue? If so, what document selectors are you using for the different workspace folders?

[vinistock]

Yes, it still happens. In fact, we started noticing that code snippets in Copilot Chat also trigger a semantic highlighting request before ever sending a textDocument/didOpen notification. It kept erroring for us because the server wasn't aware of that document's existence.

Just recently, we tried to limit our document selector to only the file and git schemes (PR), but we still see the Copilot Chat window firing semantic highlighting requests.

Our document selector is configured like this (simplified):

const selector = [
  // Selectors for documents inside the workspace for `ruby` and `erb`
  { scheme: "file", language: "ruby", pattern: `${workspaceFolder.uri.fsPath}/**/*` },
  { scheme: "git", language: "ruby", pattern: `${workspaceFolder.uri.fsPath}/**/*` },
  { scheme: "file", language: "erb", pattern: `${workspaceFolder.uri.fsPath}/**/*` },
  { scheme: "git", language: "erb", pattern: `${workspaceFolder.uri.fsPath}/**/*` },

  // Selectors for unsaved files
  { scheme: "untitled", language: "ruby" },
  { scheme: "untitled", language: "erb" },

  // Selectors for bundled gem dependencies
  { scheme: "file", language: "ruby", pattern: `${home}/.gem/ruby/${rubyVersion}/gems/**/*` },

  // Selectors for default gem dependencies
  { scheme: "file", language: "ruby", pattern: `${home}/.rubies/${rubyVersion}/lib/gems/**/*` },
]

Each workspace can have its own set of dependencies and it's possible to configure their installation path completely differently. For example, you can have a workspace that installs all dependencies under ~/.gem and another one that installs inside the workspace under vendor/bundle.

However, we see these extra semantic highlighting requests for files inside the workspaces or for the Copilot Chat snippets, not for files in dependencies.

[dbaeumer]

Regarding textDocument/didOpen and request: according to the spec it is legal to send requests for documents that are not open. Open is a bad name since it is more about ownership.

I will look into why the sematic color request is not scoped correctly

[dbaeumer]

I tried to reproduce this but failed. Can you check in semanticTokens.js around here:

that this is called twice with different document selectors. Can you also let me know how these selectors look like.

[vinistock]

Okay, I just debugged in the place you mentioned and I got the exact values for what we declare as document selectors and what reaches that point in the code.

The document selector we pass as a client option, has the following entries.

[
  // ERB files inside the workspace for file and git
  { scheme: 'file', language: 'erb', pattern: '/workspace/path/ruby-lsp/**/*' },
  { scheme: 'git', language: 'erb', pattern: '/workspace/path/ruby-lsp/**/*' },
  
  // Ruby files inside the workspace for file and git
  { scheme: 'file', language: 'ruby', pattern: '/workspace/path/ruby-lsp/**/*' },
  { scheme: 'git', language: 'ruby', pattern: '/workspace/path/ruby-lsp/**/*' },

  // Ruby and ERB files for the untitled scheme
  { scheme: 'untitled', language: 'ruby' },
  { scheme: 'untitled', language: 'erb' },

  // Ruby files in bundled gems
  { scheme: 'file', language: 'ruby', pattern: '$HOME/.gem/ruby/3.4.1/**/*' },
  { scheme: 'git', language: 'ruby', pattern: '$HOME/.gem/ruby/3.4.1/**/*' },

  // Ruby files in default gems
  { scheme: 'file', language: 'ruby', pattern: '/RUBY_PATH/3.4.1/lib/ruby/gems/3.4.0/**/*' },
  { scheme: 'git', language: 'ruby', pattern: '/RUBY_PATH/3.4.1/lib/ruby/gems/3.4.0/**/*' },

  // Ruby files in bundled gems stored inside of the Ruby installation
  { scheme: 'file', language: 'ruby', pattern: '/RUBY_PATH/3.4.1/lib/ruby/3.4.0/**/*' },
  { scheme: 'git', language: 'ruby', pattern: '/RUBY_PATH/3.4.1/lib/ruby/3.4.0/**/*' },
]

Debugging in the spot that you indicated, registerLanguageProvider is invoked only once when a single workspace is opened in VS Code, but the value of const selector = options.documentSelector; is the following:

[
  { language: "ruby" },
  { language: "erb" }
]

Could the lack of the schemes and patterns be the reason here? If I ask Copilot Chat to solve a simple problem in Ruby, I see the code blocks in the chat fire a bunch of requests for vscode-chat-code-block:// scheme URIs (despite we not including that in the selector).

Let me know if I can provide more information to help.

[dbaeumer]

@vinistock do you by any chance specify a document selector on the server side when returning from the initialize request? Newer requests (like semantic tokens) support that and it would override the client side default selector.

[dbaeumer]

The decision is made here: https://github.com/microsoft/vscode-languageserver-node/blob/dbaeumer/lexical-skunk-aquamarine/client/src/common/client.ts#L2120

[vinistock]

You once again cleared things up. It was indeed the server that was incorrectly overriding our document selectors and not scoping it to any scheme or patterns. This is fixed in Shopify/ruby-lsp#3074. Thank you very much for the help, I appreciate it.

Do you believe it would make sense for the client package to emit a warning if the original document selector was overridden by the server?

Also, please feel free to close this issue.

[dbaeumer]

What we could add is a trace entry saying under which document selector a feature got registered. Making this a warning is IMO to hard since servers might really do this on purpose.

[vinistock]

Yeah, that makes sense. Would you like to have that added? I can put a PR up if you think it's worth doing it.

[dbaeumer]

PRs are always welcome!

[vinistock]

I looked a little bit into this. I think the spot where server capabilities are assigned is where the selectors are overridden for static registration. Is that correct?

Logging something in this case would require going over all registered features that support text document registration options and checking if they passed a document selector, which may honestly not be worth it.