Include Bzlmod globals in builtin.proto

[fmeum]

This allows consumers of the proto to learn about globals only available in MODULE.bazel under a new ApiContext MODULE.

Also adds the ApiContext.BZL globals for repository rule and module extension definition.

A new smoke test verifies that common symbols are contained in the proto for each API context.

Work towards bazel-contrib/vscode-bazel#1

[fmeum]

FYI @withered-magic

@meteorcloudy Not sure who to assign for this part of the codebase, feel free to reassign.

[fmeum]

Need to figure out how to add parameters.

Parameters are now available.

[fmeum]

I pushed a new commit to add module_extension and repository_rule, which were also missing.

[meteorcloudy]

@tetromino are you the right person to review this? It's not too complicated and looks fine to me.

[fmeum]

@tetromino Friendly ping

[fmeum]

@tetromino Gentle ping, following up on #21936 (comment)

[fmeum]

@tetromino Friendly ping

[vogelsgesang]

Hi @tetromino,

Thanks for sharing your thoughts as well as an alternative design earlier this year, and for capturing our feedback on additional requirements for the LSP use cases in #21979 🙂

I still agree with the overall design: aligning the generation of documentation pages and language servers to both rely on Stardoc protos is a very good solution. This makes sure that the published documentation and the in-editor suggestions from the LSP will stay in sync.

I wonder if you can share additional details on your envisioned design, timeline, roadmap by now? Maybe the opensource community could contribute somehow, if we know what you are working towards and which design and work items you are having in mind?

As one of the maintainers of the Bazel VSCode extension, I want to integrate a language server directly into the VSCode extension, such that all users can benefit from auto-completion, signature help etc. However, with the current state of both language servers (bazel-lsp by @cameron-martin and starpls by @withered-magic), I don't feel comfortable shipping them to a larger user base. We would immediately get bug reports that autocompletion only works unreliably or even not at all for bzlmod files. As such, I need your help to make this happen 🙂

I guess that Google has its own, internal language server for Starlark / Bazel and hence I understand that this is not your highest priority. If you could share your roadmap, envisioned design etc. we, the community, could maybe fill in the blanks to pave the path towards a widely available Bazel language server.

Alternatively (e.g., in case the stardoc project is not getting sufficient funding inside Google for whatever reason), I would appreciate if we could reconsider merging this pull request as well as #21135, at least as a stop gap solution until we can work towards the long-term stardoc-based solution

[tetromino]

Hi @vogelsgesang

I wonder if you can share additional details on your envisioned design, timeline, roadmap by now? Maybe the opensource community could contribute somehow, if we know what you are working towards and which design and work items you are having in mind?

Sure.

Step zero was a series of refactorings and cleanups around stardoc_output.proto, which was finished by mid-May.

Step one was adding a way to represent the syntactic role of function params (fixed by bd1c3af) so that a documentation renderer can output *args, *, and **kwargs correctly in all cases.

Steps two and three will be to represent 2 additional bits of data in stardoc_output.proto:

• type information for FunctionParamInfo and FunctionReturnInfo (see Stardoc protos and ModuleInfoExtractor / starlark_doc_extract should export parameter type annotations for builtin functions #21979)
• a message representing a documented non-callable symbol (see Docstrings for top-level constant values (strings, lists, dicts, structs) stardoc#202)

I hope get this done in early July.

[vogelsgesang]

Thanks for the update and the rough timeline!

Are the remaining steps 2 and 3 all that's left to be done?

Afaik, there is currently no way to dump Bazel's builtin functions, yet. From the Stardoc documentation, my understanding is that it is only possible to run stardoc on Starlark files? But Bazel's builtins are not all migrated to Starlark, and some of them probably never will.

Is there still some additional command (e.g., bazel info builtin-symbols-as-proto, similar to #21936) missing? Or is this also covered by #21979?