Merge remote-tracking branch 'upstream/main' into pk-add-parallel-ten…

…ant-creation-test

.github/ISSUE_TEMPLATE/DESIGN_DOCUMENT_TEMPLATE.md

@@ -0,0 +1,140 @@
+- [Using this Document](#using-this-document)
+	- [Feature Description](#feature-description-)
+	- [Architecture Diagram](#architecture-diagram)
+		- [API Diagram](#api-diagram)
+	- [Documenting your Feature](#documenting-your-feature)
+	- [Security Reviews](#security-reviews)
+
+## Document Checklist
+- [ ] Architecture diagram created 
+- [ ] API diagram created
+- [ ] Documentation pull request opened
+- [ ] Security review opened if required
+
+<!--
+# Using this Document
+
+This document provides an easy-to-use template to help you get started building your own design document for OpenSearch features.
+
+To use this document you should:
+
+1. Copy and paste the contents of the document to a new file <YOUR_FEATURE>.md or to the body of a GitHub comment.
+2. Fill out the remainder of this document.
+
+**Please do not overwrite this document.**
+
+-->
+
+## Feature Description
+
+<!--
+_Please provide a **brief** description of what feature you are introducing and what it does. This should include information about the scope of the change and and its technical aspects. It should not reiterate the GitHub description._
+
+For example: This feature creates a token with properties x, y, & z, it will have Rest APIs: POST /_security/foo/token { ... }. It will add functionality in class files AFactory, BSingleton, and CInstance.
+-->
+
+This document outlines a new feature <YOUR_FEATURE>.
+
+<YOUR_FEATURE> is...
+
+## Architecture Diagram
+
+
+
+<!--_Some features are best explained using architecture diagrams. In particular, Mermaid diagrams are supported by GitHub and preferred. You can find examples of Mermaid diagrams in the [ARCHITECTURE.md](./ARCHITECTURE.md) file._-->
+
+<!--Here is a generic graph diagram you can modify:-->
+
+```mermaid
+graph TD
+    subgraph Subgraph
+        subgraph Subsubgraph1
+            l1[Label 1]
+            l2[Label 2]
+        end
+        subgraph Subsubgraph2
+            l3(Label 3)
+            l4[Label 4]
+        end
+        subgraph Subsubgraph3
+           l5(Label 5)
+           l6[Label 6]
+        end
+
+        l1 -- Line 1 --> l3
+        l2 -- Line 2 --> l4
+        l3 -- Line 3 --> l5
+    end
+```
+
+<!--Similarly, this is a sample sequence diagram:-->
+
+```mermaid
+sequenceDiagram
+	title Sample Sequence
+	autonumber
+
+    A->>B: Request 1
+    B->>C: Request 1 Received
+    C->>D: Handle Request 
+    D->>C: Do Something
+    C-->>B: Allow/Deny request
+    C->>E: Update Audit Log
+    D-->>B: Result
+    B->>A: Response
+```
+
+<!--There are several other types of diagrams also supported by Mermaid.-->
+
+
+### API Diagram
+
+<!--If your change introduces new API routes, please provide a diagram of the changes.-->
+
+<!--The sequence diagram is recommended for this purpose:-->
+
+```mermaid
+sequenceDiagram
+	title Sample Sequence
+	autonumber
+
+    A->>B: Request 1
+    B->>C: Request 1 Received
+    C->>D: Handle Request 
+    D->>C: Do Something
+    C-->>B: Allow/Deny request
+    C->>E: Update Audit Log
+    D-->>B: Result
+    B->>A: Response
+```
+
+## Documenting your Feature
+
+<!--Whenever you are making a large change, you should make sure you have created the appropriate updates to the documentation website.
+
+Please use this section as a reminder to submit a pull request to the documentation repository with the details of your change.
+
+If you do not complete this step, your pull request may not be merged.
+
+Please link your documentation pull request here when it is complete and check the box below.-->
+
+## Security Reviews
+
+Certain changes require advanced security reviews by parties which depend on OpenSearch.
+
+As part of this process, we request that the following decision tree is used to determine in which cases an advanced security review is required.
+
+In cases where an advanced review is required, additional time is needed to merge pull requests. In these cases, the earlier pertinent information is provided the better.
+
+Please denote whether a review is expected to be required.
+
+```mermaid
+graph TD
+  A[Is it a new feature or a feature update?] -->|New Feature| B[Does the feature handle sensitive data?]
+  A -->|Feature Update| C[Does the update change security-critical components?]
+  B -->|Yes| D[Create a security review]
+  B -->|No| E[Do a basic security assessment]
+  C -->|Yes| D[Create a security review]
+  C -->|No| E[Do a basic security assessment]
+  E -->|Security concern found?| D[Create a security review]
+```

.github/actions/run-bwc-suite/action.yaml

@@ -41,16 +41,16 @@ runs:
       with:
         cache-disabled: true
         arguments: |
+          -p bwc-test
           bwcTestSuite
           -Dtests.security.manager=false
           -Dtests.opensearch.secure=true
           -Dtests.opensearch.username=${{ inputs.username }}
           -Dtests.opensearch.password=${{ inputs.password }}
           -Dbwc.version.previous=${{ steps.build-previous.outputs.built-version }}
           -Dbwc.version.next=${{ steps.build-next.outputs.built-version }} -i
-        build-root-directory: bwc-test
 
-    - uses: actions/upload-artifact@v3
+    - uses: alehechka/upload-tartifact@v2
       if: always()
       with:
         name: ${{ inputs.report-artifact-name }}

.github/dependabot.yml

@@ -4,8 +4,6 @@ updates:
     directory: "/"
     schedule:
       interval: "weekly"
-    commit-message:
-      prefix: "dependabot:"
     ignore:
       # For all packages, ignore all major versions to minimize breaking issues
       - dependency-name: "*"
@@ -14,5 +12,3 @@ updates:
     directory: "/"
     schedule:
       interval: "weekly"
-    commit-message:
-      prefix: "dependabot:"

.github/workflows/automatic-merges.yml

@@ -19,7 +19,7 @@ jobs:
         with:
           token: ${{ secrets.GITHUB_TOKEN }}
 
-      - uses: peternied/discerning-merger@v1
+      - uses: peternied/discerning-merger@v3
         if: steps.find-triggering-pr.outputs.pr-number != null
         with:
           token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/ci.yml

@@ -40,7 +40,7 @@ jobs:
       matrix:
         gradle_task: ${{ fromJson(needs.generate-test-list.outputs.separateTestsNames) }}
         platform: [windows-latest, ubuntu-latest]
-        jdk: [11, 17]
+        jdk: [11, 17, 21]
     runs-on: ${{ matrix.platform }}
 
     steps:
@@ -60,34 +60,57 @@ jobs:
         arguments: |
           ${{ matrix.gradle_task }} -Dbuild.snapshot=false
 
-    - name: Coverage
-      uses: Wandalen/[email protected]
-      with:
-        attempt_limit: 3
-        attempt_delay: 2000
-        action: codecov/codecov-action@v3
-        with: |
-          token: ${{ secrets.CODECOV_TOKEN }}
-          fail_ci_if_error: false
-          files: ./build/reports/jacoco/test/jacocoTestReport.xml
-
-    - uses: actions/upload-artifact@v3
+    - uses: alehechka/upload-tartifact@v2
       if: always()
       with:
-        name: ${{ matrix.platform }}-JDK${{ matrix.jdk }}-reports
+        name: ${{ matrix.platform }}-JDK${{ matrix.jdk }}-${{ matrix.gradle_task }}-reports
         path: |
           ./build/reports/
 
-    - name: check archive for debugging
-      if: always()
-      run: echo "Check the artifact ${{ matrix.platform }}-JDK${{ matrix.jdk }}-reports for detailed test results"
+  report-coverage:
+    needs:
+      - "test"
+      - "integration-tests"
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/download-artifact@v3
+        with:
+          path: downloaded-artifacts
+
+      - name: Display structure of downloaded files
+        run: ls -R
+        working-directory: downloaded-artifacts
+
+      - name: Extract downloaded artifacts
+        run: |
+          for archive in ./*/artifact.tar; do
+            (cd "$(dirname "$archive")" && tar -xvf artifact.tar)
+          done
+        working-directory: downloaded-artifacts
+
+      - name: Display structure of downloaded files
+        run: ls -R
+        working-directory: downloaded-artifacts
+
+      - name: Upload Coverage with retry
+        uses: Wandalen/[email protected]
+        with:
+          attempt_limit: 5
+          attempt_delay: 2000
+          action: codecov/codecov-action@v3
+          with: |
+            token: ${{ secrets.CODECOV_TOKEN }}
+            fail_ci_if_error: true
+            directory: downloaded-artifacts
+            verbose: true
+
 
   integration-tests:
     name: integration-tests
     strategy:
       fail-fast: false
       matrix:
-        jdk: [11, 17]
+        jdk: [11, 17, 21]
         platform: [ubuntu-latest] # Removed windows https://github.com/opensearch-project/security/issues/3423
     runs-on: ${{ matrix.platform }}
 
@@ -108,6 +131,14 @@ jobs:
         arguments: |
           integrationTest -Dbuild.snapshot=false
 
+    - uses: alehechka/upload-tartifact@v2
+      if: always()
+      with:
+        name: integration-${{ matrix.platform }}-JDK${{ matrix.jdk }}-reports
+        path: |
+          ./build/reports/
+
+
   resource-tests:
     env:
       CI_ENVIRONMENT: resource-test

.github/workflows/integration-tests.yml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        jdk: [11, 17]
+        jdk: [11, 17, 21]
         test-run: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 
     steps:
@@ -24,7 +24,7 @@ jobs:
 
     - run: OPENDISTRO_SECURITY_TEST_OPENSSL_OPT=true ./gradlew test
 
-    - uses: actions/upload-artifact@v3
+    - uses: alehechka/upload-tartifact@v2
       if: always()
       with:
         name: ${{ matrix.jdk }}-${{ matrix.test-run }}-reports

.github/workflows/plugin_install.yml

@@ -12,7 +12,7 @@ jobs:
       fail-fast: false
       matrix:
         os: [ubuntu-latest, windows-latest]
-        jdk: [11, 17]
+        jdk: [11, 17, 21]
     runs-on: ${{ matrix.os }}
 
     steps: