generated from delphix/.github
-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove sound directory from kernel build #23
Draft
prakashsurya
wants to merge
10,000
commits into
develop
Choose a base branch
from
projects/ps-overrides
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BugLink: https://bugs.launchpad.net/bugs/1903087 The new functions use device_{online,offline}() which are userspace safe. This is in preparation to move cpu_{up, down} kernel users to use a safer interface that is not racy with userspace. Suggested-by: "Paul E. McKenney" <[email protected]> Signed-off-by: Qais Yousef <[email protected]> Signed-off-by: Thomas Gleixner <[email protected]> Reviewed-by: Paul E. McKenney <[email protected]> Link: https://lkml.kernel.org/r/[email protected] (cherry picked from commit 93ef142) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves driver handles the enclave lifetime management. This includes enclave creation, termination and setting up its resources such as memory and CPU. An enclave runs alongside the VM that spawned it. It is abstracted as a process running in the VM that launched it. The process interacts with the NE driver, that exposes an ioctl interface for creating an enclave and setting up its resources. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * No changes. v7 -> v8 * Add NE custom error codes for user space memory regions not backed by pages multiple of 2 MiB, invalid flags and enclave CID. * Add max flag value for enclave image load info. v6 -> v7 * Clarify in the ioctls documentation that the return value is -1 and errno is set on failure. * Update the error code value for NE_ERR_INVALID_MEM_REGION_SIZE as it gets in user space as value 25 (ENOTTY) instead of 515. Update the NE custom error codes values range to not be the same as the ones defined in include/linux/errno.h, although these are not propagated to user space. v5 -> v6 * Fix typo in the description about the NE CPU pool. * Update documentation to kernel-doc format. * Remove the ioctl to query API version. v4 -> v5 * Add more details about the ioctl calls usage e.g. error codes, file descriptors used. * Update the ioctl to set an enclave vCPU to not return a file descriptor. * Add specific NE error codes. v3 -> v4 * Decouple NE ioctl interface from KVM API. * Add NE API version and the corresponding ioctl call. * Add enclave / image load flags options. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * Add ioctl for getting enclave image load metadata. * Update NE_ENCLAVE_START ioctl name to NE_START_ENCLAVE. * Add entry in Documentation/userspace-api/ioctl/ioctl-number.rst for NE ioctls. * Update NE ioctls definition based on the updated ioctl range for major and minor. Reviewed-by: Alexander Graf <[email protected]> Reviewed-by: Stefan Hajnoczi <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (backported from commit 15b760c) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves (NE) driver communicates with a new PCI device, that is exposed to a virtual machine (VM) and handles commands meant for handling enclaves lifetime e.g. creation, termination, setting memory regions. The communication with the PCI device is handled using a MMIO space and MSI-X interrupts. This device communicates with the hypervisor on the host, where the VM that spawned the enclave itself runs, e.g. to launch a VM that is used for the enclave. Define the MMIO space of the NE PCI device, the commands that are provided by this device. Add an internal data structure used as private data for the PCI device driver and the function for the PCI device command requests handling. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Fix indent for the NE PCI device command types enum. v7 -> v8 * No changes. v6 -> v7 * Update the documentation to include references to the NE PCI device id and MMIO bar. v5 -> v6 * Update documentation to kernel-doc format. v4 -> v5 * Add a TODO for including flags in the request to the NE PCI device to set a memory region for an enclave. It is not used for now. v3 -> v4 * Remove the "packed" attribute and include padding in the NE data structures. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * Update path naming to drivers/virt/nitro_enclaves. * Update NE_ENABLE_OFF / NE_ENABLE_ON defines. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru-Catalin Vasile <[email protected]> Signed-off-by: Alexandru Ciobotaru <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 0a44561) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves driver keeps an internal info per each enclave. This is needed to be able to manage enclave resources state, enclave notifications and have a reference of the PCI device that handles command requests for enclave lifetime management. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Add data structure to keep references to both Nitro Enclaves misc and PCI devices. v7 -> v8 * No changes. v6 -> v7 * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. v5 -> v6 * Update documentation to kernel-doc format. * Include in the enclave memory region data structure the user space address and size for duplicate user space memory regions checks. v4 -> v5 * Include enclave cores field in the enclave metadata. * Update the vCPU ids data structure to be a cpumask instead of a list. v3 -> v4 * Add NUMA node field for an enclave metadata as the enclave memory and CPUs need to be from the same NUMA node. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * Add enclave memory regions and vcpus count for enclave bookkeeping. * Update ne_state comments to reflect NE_START_ENCLAVE ioctl naming update. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru-Catalin Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 1df6248) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves PCI device is used by the kernel driver as a means of communication with the hypervisor on the host where the primary VM and the enclaves run. It handles requests with regard to enclave lifetime. Setup the PCI device driver and add support for MSI-X interrupts. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Init the reference to the ne_pci_dev in the ne_devs data structure. v7 -> v8 * Add NE PCI driver shutdown logic. v6 -> v7 * No changes. v5 -> v6 * Update documentation to kernel-doc format. v4 -> v5 * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Update NE PCI driver name to "nitro_enclaves". v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. * Remove the WARN_ON calls. * Remove linux/bug include that is not needed. * Update static calls sanity checks. * Remove "ratelimited" from the logs that are not in the ioctl call paths. * Update kzfree() calls to kfree(). v1 -> v2 * Add log pattern for NE. * Update PCI device setup functions to receive PCI device data structure and then get private data from it inside the functions logic. * Remove the BUG_ON calls. * Add teardown function for MSI-X setup. * Update goto labels to match their purpose. * Implement TODO for NE PCI device disable state check. * Update function name for NE PCI device probe / remove. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru-Catalin Vasile <[email protected]> Signed-off-by: Alexandru Ciobotaru <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 89308c1) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves PCI device exposes a MMIO space that this driver uses to submit command requests and to receive command replies e.g. for enclave creation / termination or setting enclave resources. Add logic for handling PCI device command requests based on the given command type. Register an MSI-X interrupt vector for command reply notifications to handle this type of communication events. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * No changes. v7 -> v8 * Update function signature for submit request and retrive reply functions as they only returned 0, no error code. * Include command type value in the error logs of ne_do_request(). v6 -> v7 * No changes. v5 -> v6 * Update documentation to kernel-doc format. v4 -> v5 * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Return IRQ_NONE when interrupts are not handled. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Remove "ratelimited" from the logs that are not in the ioctl call paths. v1 -> v2 * Add log pattern for NE. * Remove the BUG_ON calls. * Update goto labels to match their purpose. * Add fix for kbuild report: https://lore.kernel.org/lkml/202004231644.xTmN4Z1z%[email protected]/ Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru-Catalin Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit ad2b698) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 In addition to the replies sent by the Nitro Enclaves PCI device in response to command requests, out-of-band enclave events can happen e.g. an enclave crashes. In this case, the Nitro Enclaves driver needs to be aware of the event and notify the corresponding user space process that abstracts the enclave. Register an MSI-X interrupt vector to be used for this kind of out-of-band events. The interrupt notifies that the state of an enclave changed and the driver logic scans the state of each running enclave to identify for which this notification is intended. Create an workqueue to handle the out-of-band events. Notify user space enclave process that is using a polling mechanism on the enclave fd. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the reference to the pdev directly from the ne_pci_dev instead of the one from the enclave data structure. v7 -> v8 * No changes. v6 -> v7 * No changes. v5 -> v6 * Update documentation to kernel-doc format. v4 -> v5 * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Return IRQ_NONE when interrupts are not handled. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Remove "ratelimited" from the logs that are not in the ioctl call paths. v1 -> v2 * Add log pattern for NE. * Update goto labels to match their purpose. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru-Catalin Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit e5d616d) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 The Nitro Enclaves driver provides an ioctl interface to the user space for enclave lifetime management e.g. enclave creation / termination and setting enclave resources such as memory and CPU. This ioctl interface is mapped to a Nitro Enclaves misc device. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE misc device in the NE PCI device driver logic. v7 -> v8 * Add define for the CID of the primary / parent VM. * Update the NE PCI driver shutdown logic to include misc device deregister. v6 -> v7 * Set the NE PCI device the parent of the NE misc device to be able to use it in the ioctl logic. * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. v5 -> v6 * Remove the ioctl to query API version. * Update documentation to kernel-doc format. v4 -> v5 * Update the size of the NE CPU pool string from 4096 to 512 chars. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Remove the NE CPU pool init during kernel module loading, as the CPU pool is now setup at runtime, via a sysfs file for the kernel parameter. * Add minimum enclave memory size definition. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. * Remove the WARN_ON calls. * Remove linux/bug and linux/kvm_host includes that are not needed. * Remove "ratelimited" from the logs that are not in the ioctl call paths. * Remove file ops that do nothing for now - open and release. v1 -> v2 * Add log pattern for NE. * Update goto labels to match their purpose. * Update ne_cpu_pool data structure to include the global mutex. * Update NE misc device mode to 0660. * Check if the CPU siblings are included in the NE CPU pool, as full CPU cores are given for the enclave(s). Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit bd47c99) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add ioctl command logic for enclave VM creation. It triggers a slot allocation. The enclave resources will be associated with this slot and it will be used as an identifier for triggering enclave run. Return a file descriptor, namely enclave fd. This is further used by the associated user space enclave process to set enclave resources and trigger enclave termination. The poll function is implemented in order to notify the enclave process when an enclave exits without a specific enclave termination command trigger e.g. when an enclave crashes. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE PCI device. v7 -> v8 * No changes. v6 -> v7 * Use the NE misc device parent field to get the NE PCI device. * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. v5 -> v6 * Update the code base to init the ioctl function in this patch. * Update documentation to kernel-doc format. v4 -> v5 * Release the reference to the NE PCI device on create VM error. * Close enclave fd on copy_to_user() failure; rename fd to enclave fd while at it. * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. * Remove log on copy_to_user() failure. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Update the NE ioctl call to match the decoupling from the KVM API. * Add metadata for the NUMA node for the enclave memory and CPUs. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Update kzfree() calls to kfree(). * Remove file ops that do nothing for now - open. v1 -> v2 * Add log pattern for NE. * Update goto labels to match their purpose. * Remove the BUG_ON calls. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 38907e1) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 An enclave, before being started, has its resources set. One of its resources is CPU. A NE CPU pool is set and enclave CPUs are chosen from it. Offline the CPUs from the NE CPU pool during the pool setup and online them back during the NE CPU pool teardown. The CPU offline is necessary so that there would not be more vCPUs than physical CPUs available to the primary / parent VM. In that case the CPUs would be overcommitted and would change the initial configuration of the primary / parent VM of having dedicated vCPUs to physical CPUs. The enclave CPUs need to be full cores and from the same NUMA node. CPU 0 and its siblings have to remain available to the primary / parent VM. Add ioctl command logic for setting an enclave vCPU. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE PCI device. v7 -> v8 * No changes. v6 -> v7 * Check for error return value when setting the kernel parameter string. * Use the NE misc device parent field to get the NE PCI device. * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. * Calculate the number of threads per core and not use smp_num_siblings that is x86 specific. v5 -> v6 * Check CPUs are from the same NUMA node before going through CPU siblings during the NE CPU pool setup. * Update documentation to kernel-doc format. v4 -> v5 * Set empty string in case of invalid NE CPU pool. * Clear NE CPU pool mask on pool setup failure. * Setup NE CPU cores out of the NE CPU pool. * Early exit on NE CPU pool setup if enclave(s) already running. * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. * Add check for maximum vCPU id possible before looking into the CPU pool. * Remove log on copy_from_user() / copy_to_user() failure and on admin capability check for setting the NE CPU pool. * Update the ioctl call to not create a file descriptor for the vCPU. * Split the CPU pool usage logic in 2 separate functions - one to get a CPU from the pool and the other to check the given CPU is available in the pool. v3 -> v4 * Setup the NE CPU pool at runtime via a sysfs file for the kernel parameter. * Check enclave CPUs to be from the same NUMA node. * Use dev_err instead of custom NE log pattern. * Update the NE ioctl call to match the decoupling from the KVM API. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Update kzfree() calls to kfree(). * Remove file ops that do nothing for now - open, ioctl and release. v1 -> v2 * Add log pattern for NE. * Update goto labels to match their purpose. * Remove the BUG_ON calls. * Check if enclave state is init when setting enclave vCPU. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit ff8a4d3) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Before setting the memory regions for the enclave, the enclave image needs to be placed in memory. After the memory regions are set, this memory cannot be used anymore by the VM, being carved out. Add ioctl command logic to get the offset in enclave memory where to place the enclave image. Then the user space tooling copies the enclave image in the memory using the given memory offset. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * No changes. v7 -> v8 * Add custom error code for incorrect enclave image load info flag. v6 -> v7 * No changes. v5 -> v6 * Check for invalid enclave image load flags. v4 -> v5 * Check for the enclave not being started when invoking this ioctl call. * Remove log on copy_from_user() / copy_to_user() failure. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Set enclave image load offset based on flags. * Update the naming for the ioctl command from metadata to info. v2 -> v3 * No changes. v1 -> v2 * New in v2. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 988b7a4) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Another resource that is being set for an enclave is memory. User space memory regions, that need to be backed by contiguous memory regions, are associated with the enclave. One solution for allocating / reserving contiguous memory regions, that is used for integration, is hugetlbfs. The user space process that is associated with the enclave passes to the driver these memory regions. The enclave memory regions need to be from the same NUMA node as the enclave CPUs. Add ioctl command logic for setting user space memory region for an enclave. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE PCI device. v7 -> v8 * Add early check, while getting user pages, to be multiple of 2 MiB for the pages that back the user space memory region. * Add custom error code for incorrect user space memory region flag. * Include in a separate function the sanity checks for each page of the user space memory region. v6 -> v7 * Update check for duplicate user space memory regions to cover additional possible scenarios. v5 -> v6 * Check for max number of pages allocated for the internal data structure for pages. * Check for invalid memory region flags. * Check for aligned physical memory regions. * Update documentation to kernel-doc format. * Check for duplicate user space memory regions. * Use directly put_page() instead of unpin_user_pages(), to match the get_user_pages() calls. v4 -> v5 * Add early exit on set memory region ioctl function call error. * Remove log on copy_from_user() failure. * Exit without unpinning the pages on NE PCI dev request failure as memory regions from the user space range may have already been added. * Add check for the memory region user space address to be 2 MiB aligned. * Update logic to not have a hardcoded check for 2 MiB memory regions. v3 -> v4 * Check enclave memory regions are from the same NUMA node as the enclave CPUs. * Use dev_err instead of custom NE log pattern. * Update the NE ioctl call to match the decoupling from the KVM API. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Update kzfree() calls to kfree(). v1 -> v2 * Add log pattern for NE. * Update goto labels to match their purpose. * Remove the BUG_ON calls. * Check if enclave max memory regions is reached when setting an enclave memory region. * Check if enclave state is init when setting an enclave memory region. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 7dc9d43) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 After all the enclave resources are set, the enclave is ready for beginning to run. Add ioctl command logic for starting an enclave after all its resources, memory regions and CPUs, have been set. The enclave start information includes the local channel addressing - vsock CID - and the flags associated with the enclave. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE PCI device. v7 -> v8 * Add check for invalid enclave CID value e.g. well-known CIDs and parent VM CID. * Add custom error code for incorrect flag in enclave start info and invalid enclave CID. v6 -> v7 * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. v5 -> v6 * Check for invalid enclave start flags. * Update documentation to kernel-doc format. v4 -> v5 * Add early exit on enclave start ioctl function call error. * Move sanity checks in the enclave start ioctl function, outside of the switch-case block. * Remove log on copy_from_user() / copy_to_user() failure. v3 -> v4 * Use dev_err instead of custom NE log pattern. * Update the naming for the ioctl command from metadata to info. * Check for minimum enclave memory size. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. v1 -> v2 * Add log pattern for NE. * Check if enclave state is init when starting an enclave. * Remove the BUG_ON calls. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 111c775) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 An enclave is associated with an fd that is returned after the enclave creation logic is completed. This enclave fd is further used to setup enclave resources. Once the enclave needs to be terminated, the enclave fd is closed. Add logic for enclave termination, that is mapped to the enclave fd release callback. Free the internal enclave info used for bookkeeping. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Use the ne_devs data structure to get the refs for the NE PCI device. v7 -> v8 * No changes. v6 -> v7 * Remove the pci_dev_put() call as the NE misc device parent field is used now to get the NE PCI device. * Update the naming and add more comments to make more clear the logic of handling full CPU cores and dedicating them to the enclave. v5 -> v6 * Update documentation to kernel-doc format. * Use directly put_page() instead of unpin_user_pages(), to match the get_user_pages() calls. v4 -> v5 * Release the reference to the NE PCI device on enclave fd release. * Adapt the logic to cpumask enclave vCPU ids and CPU cores. * Remove sanity checks for situations that shouldn't happen, only if buggy system or broken logic at all. v3 -> v4 * Use dev_err instead of custom NE log pattern. v2 -> v3 * Remove the WARN_ON calls. * Update static calls sanity checks. * Update kzfree() calls to kfree(). v1 -> v2 * Add log pattern for NE. * Remove the BUG_ON calls. * Update goto labels to match their purpose. * Add early exit in release() if there was a slot alloc error in the fd creation path. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 9c8eb50) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add kernel config entry for Nitro Enclaves, including dependencies. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * No changes. v7 -> v8 * No changes. v6 -> v7 * Remove, for now, the dependency on ARM64 arch. x86 is currently supported, with Arm to come afterwards. The NE kernel driver can be built for aarch64 arch. v5 -> v6 * No changes. v4 -> v5 * Add arch dependency for Arm / x86. v3 -> v4 * Add PCI and SMP dependencies. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * Update path to Kconfig to match the drivers/virt/nitro_enclaves directory. * Update help in Kconfig. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 07499cc) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add Makefile for the Nitro Enclaves driver, considering the option set in the kernel config. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Remove -Wall flags, could use W=1 as an option for this. v7 -> v8 * No changes. v6 -> v7 * No changes. v5 -> v6 * No changes. v4 -> v5 * No changes. v3 -> v4 * No changes. v2 -> v3 * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * Update path to Makefile to match the drivers/virt/nitro_enclaves directory. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit 0f5c7b7) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add a user space sample for the usage of the ioctl interface provided by the Nitro Enclaves driver. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * No changes. v7 -> v8 * Track NE custom error codes for invalid page size, invalid flags and enclave CID. * Update the heartbeat logic to have a listener fd first, then start the enclave and then accept connection to get the heartbeat. * Update the reference link to the hugetlb documentation. v6 -> v7 * Track POLLNVAL as poll event in addition to POLLHUP. v5 -> v6 * Remove "rc" mentioning when printing errno string. * Remove the ioctl to query API version. * Include usage info for NUMA-aware hugetlb configuration. * Update documentation to kernel-doc format. * Add logic for enclave image loading. v4 -> v5 * Print enclave vCPU ids when they are created. * Update logic to map the modified vCPU ioctl call. * Add check for the path to the enclave image to be less than PATH_MAX. * Update the ioctl calls error checking logic to match the NE specific error codes. v3 -> v4 * Update usage details to match the updates in v4. * Update NE ioctl interface usage. v2 -> v3 * Remove the include directory to use the uapi from the kernel. * Remove the GPL additional wording as SPDX-License-Identifier is already in place. v1 -> v2 * New in v2. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Alexandru Vasile <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit acc4229) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add documentation on the overview of Nitro Enclaves. Include it in the virtualization specific directory. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Move the Nitro Enclaves documentation to the "virt" directory and add an entry for it in the corresponding index file. v7 -> v8 * Add info about the primary / parent VM CID value. * Update reference link for huge pages. * Add reference link for the x86 boot protocol. * Add license mention and update doc title / chapter formatting. v6 -> v7 * No changes. v5 -> v6 * No changes. v4 -> v5 * No changes. v3 -> v4 * Update doc type from .txt to .rst. * Update documentation based on the changes from v4. v2 -> v3 * No changes. v1 -> v2 * New in v2. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (backported from commit bf15d79) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903087 Add entry in the MAINTAINERS file for the Nitro Enclaves files such as the documentation, the header files, the driver itself and the user space sample. Changelog v9 -> v10 * Update commit message to include the changelog before the SoB tag(s). v8 -> v9 * Update the location of the documentation, as it has been moved to the "virt" directory. v7 -> v8 * No changes. v6 -> v7 * No changes. v5 -> v6 * No changes. v4 -> v5 * No changes. v3 -> v4 * No changes. v2 -> v3 * Update file entries to be in alphabetical order. v1 -> v2 * No changes. Reviewed-by: Alexander Graf <[email protected]> Signed-off-by: Andra Paraschiv <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Greg Kroah-Hartman <[email protected]> (cherry picked from commit e82ed73) Signed-off-by: Kamal Mostafa <[email protected]> Acked-by: Acked-by: Kleber Sacilotto de Souza <[email protected]> Acked-by: Acked-by: William Breathitt Gray <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1902281 Relax the synchronous IOMMU TLB invalidation constraint by default to get a significant performance improvement on some AWS arm64 instance types. This is a performance vs security tradeoff and it is not suitable for the generic kernel. In the particular AWS environment it is reasonable to relax the constraint without realisticly introducing security flaws. Signed-off-by: Andrea Righi <[email protected]> Acked-by: Colin Ian King <[email protected]> Acked-by: Thadeu Lima de Souza Cascardo <[email protected]> Acked-by: Kamal Mostafa <[email protected]> Signed-off-by: Kelsey Skunberg <[email protected]>
Ignore: yes Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1903158 Properties: no-test-build Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1896824 This config option has been removed by commit 973c096 upstream (vgacon: remove software scrollback support). Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
Ignore: yes Signed-off-by: Marcelo Henrique Cerri <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1907372 Properties: no-test-build Signed-off-by: Marcelo Henrique Cerri <[email protected]>
Signed-off-by: Marcelo Henrique Cerri <[email protected]>
Ignore: yes Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
BugLink: https://bugs.launchpad.net/bugs/1907586 Properties: no-test-build Signed-off-by: Kleber Sacilotto de Souza <[email protected]>
delphix-devops-bot
force-pushed
the
develop
branch
2 times, most recently
from
June 5, 2023 12:39
8e82f23
to
6dbd3ba
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
August 27, 2023 15:57
1814cea
to
6e87f48
Compare
delphix-devops-bot
force-pushed
the
develop
branch
2 times, most recently
from
September 8, 2023 15:04
4065575
to
564c071
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
September 20, 2023 14:54
b96bbd3
to
974a77a
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
October 10, 2023 14:53
974a77a
to
1e46d94
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
October 21, 2023 14:55
1e46d94
to
c31b608
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
November 1, 2023 15:10
c31b608
to
ded3266
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
November 22, 2023 16:00
ded3266
to
95274fb
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
December 10, 2023 15:30
95274fb
to
79c6b25
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
January 27, 2024 15:28
79c6b25
to
9d7a684
Compare
delphix-devops-bot
pushed a commit
that referenced
this pull request
Feb 9, 2024
BugLink: https://bugs.launchpad.net/bugs/2043422 commit 0b0747d upstream. The following processes run into a deadlock. CPU 41 was waiting for CPU 29 to handle a CSD request while holding spinlock "crashdump_lock", but CPU 29 was hung by that spinlock with IRQs disabled. PID: 17360 TASK: ffff95c1090c5c40 CPU: 41 COMMAND: "mrdiagd" !# 0 [ffffb80edbf37b58] __read_once_size at ffffffff9b871a40 include/linux/compiler.h:185:0 !# 1 [ffffb80edbf37b58] atomic_read at ffffffff9b871a40 arch/x86/include/asm/atomic.h:27:0 !# 2 [ffffb80edbf37b58] dump_stack at ffffffff9b871a40 lib/dump_stack.c:54:0 # 3 [ffffb80edbf37b78] csd_lock_wait_toolong at ffffffff9b131ad5 kernel/smp.c:364:0 # 4 [ffffb80edbf37b78] __csd_lock_wait at ffffffff9b131ad5 kernel/smp.c:384:0 # 5 [ffffb80edbf37bf8] csd_lock_wait at ffffffff9b13267a kernel/smp.c:394:0 # 6 [ffffb80edbf37bf8] smp_call_function_many at ffffffff9b13267a kernel/smp.c:843:0 # 7 [ffffb80edbf37c50] smp_call_function at ffffffff9b13279d kernel/smp.c:867:0 # 8 [ffffb80edbf37c50] on_each_cpu at ffffffff9b13279d kernel/smp.c:976:0 # 9 [ffffb80edbf37c78] flush_tlb_kernel_range at ffffffff9b085c4b arch/x86/mm/tlb.c:742:0 #10 [ffffb80edbf37cb8] __purge_vmap_area_lazy at ffffffff9b23a1e0 mm/vmalloc.c:701:0 #11 [ffffb80edbf37ce0] try_purge_vmap_area_lazy at ffffffff9b23a2cc mm/vmalloc.c:722:0 #12 [ffffb80edbf37ce0] free_vmap_area_noflush at ffffffff9b23a2cc mm/vmalloc.c:754:0 #13 [ffffb80edbf37cf8] free_unmap_vmap_area at ffffffff9b23bb3b mm/vmalloc.c:764:0 #14 [ffffb80edbf37cf8] remove_vm_area at ffffffff9b23bb3b mm/vmalloc.c:1509:0 #15 [ffffb80edbf37d18] __vunmap at ffffffff9b23bb8a mm/vmalloc.c:1537:0 #16 [ffffb80edbf37d40] vfree at ffffffff9b23bc85 mm/vmalloc.c:1612:0 #17 [ffffb80edbf37d58] megasas_free_host_crash_buffer [megaraid_sas] at ffffffffc020b7f2 drivers/scsi/megaraid/megaraid_sas_fusion.c:3932:0 #18 [ffffb80edbf37d80] fw_crash_state_store [megaraid_sas] at ffffffffc01f804d drivers/scsi/megaraid/megaraid_sas_base.c:3291:0 #19 [ffffb80edbf37dc0] dev_attr_store at ffffffff9b56dd7b drivers/base/core.c:758:0 #20 [ffffb80edbf37dd0] sysfs_kf_write at ffffffff9b326acf fs/sysfs/file.c:144:0 #21 [ffffb80edbf37de0] kernfs_fop_write at ffffffff9b325fd4 fs/kernfs/file.c:316:0 #22 [ffffb80edbf37e20] __vfs_write at ffffffff9b29418a fs/read_write.c:480:0 #23 [ffffb80edbf37ea8] vfs_write at ffffffff9b294462 fs/read_write.c:544:0 #24 [ffffb80edbf37ee8] SYSC_write at ffffffff9b2946ec fs/read_write.c:590:0 #25 [ffffb80edbf37ee8] SyS_write at ffffffff9b2946ec fs/read_write.c:582:0 #26 [ffffb80edbf37f30] do_syscall_64 at ffffffff9b003ca9 arch/x86/entry/common.c:298:0 #27 [ffffb80edbf37f58] entry_SYSCALL_64 at ffffffff9ba001b1 arch/x86/entry/entry_64.S:238:0 PID: 17355 TASK: ffff95c1090c3d80 CPU: 29 COMMAND: "mrdiagd" !# 0 [ffffb80f2d3c7d30] __read_once_size at ffffffff9b0f2ab0 include/linux/compiler.h:185:0 !# 1 [ffffb80f2d3c7d30] native_queued_spin_lock_slowpath at ffffffff9b0f2ab0 kernel/locking/qspinlock.c:368:0 # 2 [ffffb80f2d3c7d58] pv_queued_spin_lock_slowpath at ffffffff9b0f244b arch/x86/include/asm/paravirt.h:674:0 # 3 [ffffb80f2d3c7d58] queued_spin_lock_slowpath at ffffffff9b0f244b arch/x86/include/asm/qspinlock.h:53:0 # 4 [ffffb80f2d3c7d68] queued_spin_lock at ffffffff9b8961a6 include/asm-generic/qspinlock.h:90:0 # 5 [ffffb80f2d3c7d68] do_raw_spin_lock_flags at ffffffff9b8961a6 include/linux/spinlock.h:173:0 # 6 [ffffb80f2d3c7d68] __raw_spin_lock_irqsave at ffffffff9b8961a6 include/linux/spinlock_api_smp.h:122:0 # 7 [ffffb80f2d3c7d68] _raw_spin_lock_irqsave at ffffffff9b8961a6 kernel/locking/spinlock.c:160:0 # 8 [ffffb80f2d3c7d88] fw_crash_buffer_store [megaraid_sas] at ffffffffc01f8129 drivers/scsi/megaraid/megaraid_sas_base.c:3205:0 # 9 [ffffb80f2d3c7dc0] dev_attr_store at ffffffff9b56dd7b drivers/base/core.c:758:0 #10 [ffffb80f2d3c7dd0] sysfs_kf_write at ffffffff9b326acf fs/sysfs/file.c:144:0 #11 [ffffb80f2d3c7de0] kernfs_fop_write at ffffffff9b325fd4 fs/kernfs/file.c:316:0 #12 [ffffb80f2d3c7e20] __vfs_write at ffffffff9b29418a fs/read_write.c:480:0 #13 [ffffb80f2d3c7ea8] vfs_write at ffffffff9b294462 fs/read_write.c:544:0 #14 [ffffb80f2d3c7ee8] SYSC_write at ffffffff9b2946ec fs/read_write.c:590:0 #15 [ffffb80f2d3c7ee8] SyS_write at ffffffff9b2946ec fs/read_write.c:582:0 #16 [ffffb80f2d3c7f30] do_syscall_64 at ffffffff9b003ca9 arch/x86/entry/common.c:298:0 #17 [ffffb80f2d3c7f58] entry_SYSCALL_64 at ffffffff9ba001b1 arch/x86/entry/entry_64.S:238:0 The lock is used to synchronize different sysfs operations, it doesn't protect any resource that will be touched by an interrupt. Consequently it's not required to disable IRQs. Replace the spinlock with a mutex to fix the deadlock. Signed-off-by: Junxiao Bi <[email protected]> Link: https://lore.kernel.org/r/[email protected] Reviewed-by: Mike Christie <[email protected]> Cc: [email protected] Signed-off-by: Martin K. Petersen <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]> Signed-off-by: Kamal Mostafa <[email protected]> Signed-off-by: Stefan Bader <[email protected]>
delphix-devops-bot
force-pushed
the
develop
branch
from
February 9, 2024 15:31
9d7a684
to
21ea538
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
March 1, 2024 15:37
21ea538
to
b625e5d
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
March 21, 2024 16:02
b625e5d
to
3640116
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
April 20, 2024 15:08
49f9f3c
to
0c085fb
Compare
delphix-devops-bot
force-pushed
the
develop
branch
2 times, most recently
from
May 23, 2024 15:10
0f829a0
to
b0263c7
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
June 13, 2024 15:24
b0263c7
to
a5619ea
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
July 5, 2024 15:46
a5619ea
to
22f6364
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
August 6, 2024 23:02
11e22c3
to
3264826
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
August 15, 2024 16:50
1680638
to
d386ba2
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
August 23, 2024 15:50
d386ba2
to
b883edd
Compare
prakashsurya
force-pushed
the
develop
branch
from
September 25, 2024 16:29
b883edd
to
5c74cf8
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
October 21, 2024 18:35
5c74cf8
to
265e018
Compare
delphix-devops-bot
force-pushed
the
develop
branch
from
November 10, 2024 17:30
265e018
to
c48e503
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.