Merge pull request #14207 from github/felicitymay-11595-VS-code-docs

Updates to the Java and VS Code docs

docs/codeql/codeql-for-visual-studio-code/index.rst

@@ -37,7 +37,10 @@ The CodeQL extension for Visual Studio Code adds rich language support for CodeQ
   CodeQL queries using the Visual Studio Code extension.
 
 - :doc:`Working with CodeQL packs in Visual Studio Code
-  <working-with-codeql-packs-in-visual-studio-code>`: You can view and edit CodeQL packs in Visual Studio Code.
+  <working-with-codeql-packs-in-visual-studio-code>`: You can view, create, and edit all types of CodeQL pack in Visual Studio Code.
+
+- :doc:`Using the CodeQL model editor
+  <using-the-codeql-model-editor>`: You can view, create, and edit CodeQL model packs using a dedicated editor.
 
 - :doc:`Customizing settings
   <customizing-settings>`: You can edit the settings for the 
@@ -67,6 +70,7 @@ The CodeQL extension for Visual Studio Code adds rich language support for CodeQ
    running-codeql-queries-at-scale-with-mrva
    testing-codeql-queries-in-visual-studio-code
    working-with-codeql-packs-in-visual-studio-code
+   using-the-codeql-model-editor
    customizing-settings
    troubleshooting-codeql-for-visual-studio-code
    troubleshooting-variant-analysis

docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst

@@ -74,6 +74,8 @@ To use the starter workspace:
 
 #. In VS Code, use the **File** > **Open Workspace** option to open the ``vscode-codeql-starter.code-workspace`` file from your checkout of the workspace repository.
 
+Remember to update the ``ql`` submodule in the starter workspace periodically to ensure that it remains compatible with newer versions of the VS Code extension and the CodeQL CLI.
+
 .. _existing-workspace:
 
 Updating an existing workspace for CodeQL

docs/codeql/codeql-for-visual-studio-code/using-the-codeql-model-editor.rst

@@ -0,0 +1,139 @@
+:tocdepth: 1
+
+.. _using-the-codeql-model-editor:
+
+Using the CodeQL model editor
+=============================
+
+.. include:: ../reusables/beta-note-model-pack-editor-vsc.rst
+
+You can view, write, and edit CodeQL packs in Visual Studio Code using the CodeQL extension. The model editor is designed to help you model external dependencies of your codebase that are not supported by the standard CodeQL Libraries.
+
+About the CodeQL model editor
+-----------------------------
+
+The CodeQL model editor guides you through modeling the calls to external dependencies in your application or fully modeling all the public entry and exit points in an external dependency.
+
+When you open the model editor, it analyzes the currently selected CodeQL database and identifies where the application uses external APIs and all public methods. An external (or third party) API is any API that is not part of the CodeQL database you have selected.
+
+The model editor has two different modes:
+
+- Application mode (default view): The editor lists each external framework used by the selected CodeQL database. When you expand a framework, a list of all calls to and from the external API is shown with the options available to model dataflow through each call. This mode is most useful for improving the CodeQL results for the specific codebase.
+
+- Dependency mode: The editor identifies all of the publicly accessible APIs in the selected CodeQL database. This view guides you through modeling each public API that the codebase makes available. When you have finished modeling the entire API, you can save the model and use it to improve the CodeQL analysis for all codebases that use the dependency.
+
+Displaying the CodeQL model editor
+----------------------------------
+
+#. Open your CodeQL workspace in VS Code, for example, the ``vscode-codeql-starter`` workspace.
+   If you haven't updated the ``ql`` submodule for a while, update it from ``main`` to ensure that you have the queries used to gather data for the model editor.
+#. Open the CodeQL extension and select the CodeQL database that you want to model from the "Databases" section of the left side pane.
+#. Use the command palette to run the “CodeQL: Open Model Editor (Beta)” command.
+#. The CodeQL model editor will open in a new tab and run a series of telemetry queries to identify APIs in the code.
+#. When the queries are complete, the APIs that have been identified are shown in the editor.
+
+Modeling the calls your codebase makes to external APIs
+-------------------------------------------------------
+
+You typically use this approach when you are looking at a specific codebase where you want to improve the precision of CodeQL results. This is usually when the codebase uses frameworks or libraries that are not supported by CodeQL and if the source code of the framework or library is not included in the analysis.
+
+#. Select the CodeQL database that you want to improve CodeQL coverage for.
+#. Display the CodeQL model editor. By default the editor runs in application mode, so the list of external APIs used by the selected codebase is shown.
+
+   .. image:: ../images/codeql-for-visual-studio-code/model-application-mode.png
+      :width: 800
+      :alt: Screenshot of the "Application mode" view of the CodeQL model pack editor in Visual Studio Code showing three of the external frameworks used by the "sofa-jraft" codebase.
+
+#. Click to expand an external API and view the list of calls from the codebase to the external dependency.
+#. Click **View** associated with an API call or method to show where it is used in your codebase.
+
+   .. image:: ../images/codeql-for-visual-studio-code/model-application-mode-expanded.png
+      :width: 800
+      :alt: Screenshot of the "Application mode" view of the CodeQL model pack editor in Visual Studio Code showing the calls to the "rocksdbjni" framework ready for modeling. The "View" option for the first call is highlighted with a dark orange outline.
+
+#. When you have determined how to model the call or method, define the **Model type**.
+#. The remaining fields are updated with available options:
+
+   - **Source**: choose the **Output** element to model.
+   - **Sink**: choose the **Input** element to model.
+   - **Flow summary**: choose the **Input** and **Output** elements to model.
+
+#. Define the **Kind** of dataflow for the model.
+#. When you have finished modeling, click **Save all** or **Save** (shown at the bottom right of each expanded list of calls). The percentage of calls modeled in the editor is updated.
+
+The models are stored in your workspace at ``.github/codeql/extensions/<codeql-model-pack>``, where ``<codeql-model-pack>`` is the name of the CodeQL database that you selected. That is, the name of the repository, hyphen, the language analyzed by CodeQL.
+
+The models are stored in a series of YAML data extension files, one for each external API. For example:
+
+.. code-block:: none
+
+   .github/codeql/extensions/sofa-jraft-java # the model pack directory
+      models
+         jmh-core.model.yml                  # models calls to [email protected]
+         rocksdbjni.model.yml                # models calls to [email protected]
+
+Modeling the public API of a codebase
+-------------------------------------
+
+You typically use this method when you want to model a framework or library that your organization uses in more than one codebase. Once you have finished creating and testing the model, you can publish the CodeQL model pack to the GitHub Container Registry for your whole organization to use.
+
+#. Select the CodeQL database that you want to model.
+#. Display the CodeQL model editor. By default the editor runs in application mode. Click **Model as dependency** to display dependency mode. The screen changes to show the public API of the framework or library.
+
+   .. image:: ../images/codeql-for-visual-studio-code/model-dependency-mode.png
+      :width: 800
+      :alt: Screenshot of the "Dependency mode" view of the CodeQL model pack editor in Visual Studio Code showing three of the packages published by the "sofa-jraft" codebase.
+
+#. Click to expand a package and view the list of available methods.
+#. Click **View** associated with a method to show its definition.
+
+   .. image:: ../images/codeql-for-visual-studio-code/model-dependency-mode-expanded.png
+      :width: 800
+      :alt: Screenshot of the "Dependency mode" view of the CodeQL model pack editor in Visual Studio Code showing the public methods available in the "com.alipay.soft.jraft.option" package ready for modeling. The "View" option for the first method is highlighted with a dark orange outline.
+
+#. When you have determined how to model the method, define the **Model type**.
+#. The remaining fields are updated with available options:
+
+   - **Source**: choose the **Output** element to model.
+   - **Sink**: choose the **Input** element to model.
+   - **Flow summary**: choose the **Input** and **Output** elements to model.
+
+#. Define the **Kind** of dataflow for the model.
+#. When you have finished modeling, click **Save all** or **Save** (shown at the bottom right of each expanded list of calls). The percentage of calls modeled in the editor is updated.
+
+The models are stored in your workspace at ``.github/codeql/extensions/<codeql-model-pack>``, where ``<codeql-model-pack>`` is the name of the CodeQL database that you selected. That is, the name of the repository, hyphen, the language analyzed by CodeQL.
+
+The models are stored in a series of YAML data extension files, one for each public method. For example:
+
+.. code-block:: none
+
+   .github/codeql/extensions/sofa-jraft-java          # the model pack directory
+      models
+         com.alipay.sofa.jraft.option.model.yml       # models public methods in package
+         com.alipay.sofa.jraft.rhea.options.model.yml # models public methods in package
+
+The editor will create a separate model file for each package that you model.
+
+Testing CodeQL model packs
+--------------------------
+
+You can test any CodeQL model packs you create in VS Code by toggling the "use model packs" setting on and off. This method works for both databases and for variant analysis repositories.
+
+- To run queries on a CodeQL database with any model packs that are stored within the ``.github/codeql/extensions`` directory of the workspace, update your ``settings.json`` file with: ``"codeQL.runningQueries.useModelPacks": all,``
+- To run queries on a CodeQL database without using model packs, update your ``settings.json`` file with: ``"codeQL.runningQueries.useModelPacks": none,``
+
+If your model is working well, you should see a difference in the results of the two different runs. If you don't see any differences in results, you may need to introduce a known bug to verify that the model behaves as expected.
+
+Using CodeQL model packs with code scanning
+-------------------------------------------
+
+There are two methods for using CodeQL model packs with code scanning:
+
+#. Copy the model pack directory into the ``.github/codeql/extensions`` directory of the repository. It will automatically be used by all future code scanning analysis for the repository (default setup or advanced setup).
+#. Publish the model pack to the GitHub Container Registry as a CodeQL model pack. This can be downloaded and used by advanced setup for code scanning or by the CodeQL CLI running in an external CI system.
+
+For more information, see the following articles on the GitHub Docs site:
+
+- Default setup of code scanning: `Extending CodeQL coverage with CodeQL model packs in default setup <https://docs.github.com/en/code-security/code-scanning/managing-your-code-scanning-configuration/editing-your-configuration-of-default-setup#extending-codeql-coverage-with-codeql-model-packs-in-default-setup>`__
+- Advanced setup of code scanning: `Extending CodeQL coverage with CodeQL model packs <https://docs.github.com//en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning#extending-codeql-coverage-with-codeql-model-packs>`__
+- CodeQL CLI setup in external CI system: `Using model packs to analyze calls to custom dependencies <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#using-model-packs-to-analyze-calls-to-custom-dependencies>`__

docs/codeql/codeql-for-visual-studio-code/working-with-codeql-packs-in-visual-studio-code.rst

@@ -7,33 +7,49 @@ Working with CodeQL packs in Visual Studio Code
 
 .. include:: ../reusables/beta-note-package-management.rst
 
-You can view CodeQL packs and write and edit queries for them in Visual Studio Code. 
+You can view, write, and edit all types of CodeQL packs in Visual Studio Code using the CodeQL extension. 
 
 About CodeQL packs
 ------------------
-CodeQL packs are used to create, share, depend on, and run CodeQL queries and libraries. You can publish your own CodeQL packs and download packs created by others. For more information, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
+You use CodeQL packs to share your expertise in query writing, CodeQL library development, and modeling dependencies with other users. The CodeQL package management system ensures that when you publish a CodeQL pack it is ready to use, without any compilation. Anything the CodeQL pack depends on is explicitly defined within the pack. You can publish your own CodeQL packs and download packs created by others. For more information, see "`About CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/codeql-cli-reference/about-codeql-packs>`__."
 
-Using standard CodeQL packs in Visual Studio Code
---------------------------------------------------------------
+There are three types of CodeQL packs, each with a specific purpose.
+
+- Query packs are designed to be run. When a query pack is published, the bundle includes all the transitive dependencies and pre-compiled representations of each query, in addition to the query sources. This ensures consistent and efficient execution of the queries in the pack.
+- Model packs are used to model dependencies that are not supported by the standard CodeQL libraries. When you add a model pack to your analysis, all relevant queries also recognize the sources, sinks and flow steps of the dependencies defined in the pack.
+- Library packs are designed to be used by query packs (or other library packs) and do not contain queries themselves. The libraries are not compiled separately.
+
+Using the CodeQL packs shipped with the CLI in Visual Studio Code
+-----------------------------------------------------------------
 To install dependencies for a CodeQL pack in your Visual Studio Code workspace, run the **CodeQL: Install Pack Dependencies** command from the Command Palette and select the packs you want to install dependencies for.
 
 You can write and run query packs that depend on the CodeQL standard libraries, without needing to check out the standard libraries in your workspace. Instead, you can install only the dependencies required by the query packs you want to use.
 
-Creating and editing CodeQL packs in Visual Studio Code
--------------------------------------------------------
-To create a new CodeQL pack, you will need to use the CodeQL CLI from a terminal, which you can do within Visual Studio Code or outside of it with the ``codeql pack init`` command. Once you create an empty pack, you can edit the ``qlpack.yml`` file or run the ``codeql pack add`` command to add dependencies or change the name or version. For more information, see "`Creating and working with CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-and-working-with-codeql-packs>`__."
+Working with CodeQL query packs
+-------------------------------
+
+One of the main benefits of working with a CodeQL query pack is that all dependencies are resolved, not just those defined within the query and standard libraries.
+
+Creating and editing CodeQL query packs 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To create a new query pack, you will need to use the CodeQL CLI from a terminal, which you can do within Visual Studio Code or outside of it with the ``codeql pack init`` command. Once you create an empty pack, you can edit the ``qlpack.yml`` file or run the ``codeql pack add`` command to add dependencies or change the name or version. For detailed information, see "`Creating and working with CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/creating-and-working-with-codeql-packs>`__."
 
 You can create or edit queries in a CodeQL pack in Visual Studio Code as you would with any CodeQL query, using the standard code editing features such as autocomplete suggestions to find elements to use from the pack's dependencies. 
 
-You can then use the CodeQL CLI to publish your pack to share with others. For more information, see "`Publishing and using CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs>`__."
+You can then use the CodeQL CLI to publish your pack to share with others. For detailed information, see "`Publishing and using CodeQL packs <https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs>`__."
 
-Viewing CodeQL packs and their dependencies in Visual Studio Code
------------------------------------------------------------------
-To download a CodeQL pack that someone else has created, run the **CodeQL: Download Packs** command from the Command Palette.
-You can download all the core CodeQL query packs, or enter the full name of a specific pack to download. For example, to download the core queries for analyzing Java or Kotlin, enter ``codeql/java-queries``.
+Viewing CodeQL query packs and their dependencies
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+To download a query pack that someone else has created, run the **CodeQL: Download Packs** command from the Command Palette.
+You can download all the core query packs, or enter the full name of a specific pack to download. For example, to download the core queries for analyzing Java and Kotlin, enter ``codeql/java-queries``.
 
 Whether you have downloaded a CodeQL pack or created your own, you can open the ``qlpack.yml`` file in the root of a CodeQL pack directory in Visual Studio Code and view the dependencies section to see what libraries the pack depends on.
 
 If you want to understand a query in a CodeQL pack better, you can open the query file and view the code, using the IntelliSense code editing features of Visual Studio Code. For example, if you hover over an element from a library depended on by the pack, Visual Studio Code will resolve it so you can see documentation about the element. 
 
 To view the full definition of an element of a query, you can right-click and choose **Go to Definition**. If the library pack is present within the same Visual Studio Code workspace, this will take you to the definition within the workspace. Otherwise it will take you to the definition within your package cache, the shared location where downloaded dependencies are stored, which is in your home directory by default.
+
+Working with CodeQL model packs
+-------------------------------
+
+The CodeQL extension for Visual Studio Code includes a dedicated editor for creating and editing model packs. For information on using the model editor, see ":ref:`Using the CodeQL model editor <using-the-codeql-model-editor>`."