Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Centralize and update test documentation - Jenkins features reference #4690

Merged
merged 5 commits into from
Nov 22, 2023

Conversation

Ndacyayisenga-droid
Copy link
Contributor

Add Jenkins Features Reference

fixes part of #1558

cc @ShelleyLambert @hendrikebbers

@Ndacyayisenga-droid Ndacyayisenga-droid changed the title Centralize and update test documentation - JekinsFeatures.md Centralize and update test documentation - Jenkins features reference Aug 2, 2023
@@ -0,0 +1,129 @@
# Jenkins Features Reference
The "Jenkins Features Reference" is a comprehensive documentation outlining the various [configuration parameters](https://ci.adoptium.net/view/Test_grinder/job/Grinder/build?delay=0sec) and options available for running [Grinder jobs](https://ci.adoptium.net/view/Test_grinder/job/Grinder/) on the Jenkins server.
It serves as a guide for testers and developers using AQAvit Grinder to perform testing and triage on the ci.adoptium.net Jenkins server and the OpenJ9 Jenkins server (and internal servers).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you convert ci.adoptium.net to a link and add a link to the OpenJ9 Jenkins server?

doc/pages/JenkinFeatures.md Outdated Show resolved Hide resolved
### Test Repositories Parameters
Repositories where we pull test material from. Unless you are testing test code, these do not need to be changed.

- `ADOPTOPENJDK_REPO`: optional: your fork of aqa-tests.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume a table would make sense (see https://docs.github.com/de/get-started/writing-on-github/working-with-advanced-formatting/organizing-information-with-tables)

Next to this I would really prefer to have a sample for each property to see in what form an repo need to be specified. Is it "foo/bar" or "https://github.com/foo/bar" for example

Additional test repositories that can be overlaid onto existing AQA test material for extra testing.
This is used for laying down smoke tests in the temurin-build repository and installer tests in the installer repository, along with any private vendor tests that cannot be run in the open.

- `VENDOR_TEST_REPOS`: optional, comma-separated list of repositories containing additional test material.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again: table + samples

Comment on lines +6 to +7
## Access Permissions
For access permissions to the Grinder job at ci.adoptium.net, you need to be part of the `test-triage` Github team (requires [2FA](https://docs.github.com/en/authentication/securing-your-account-with-two-factor-authentication-2fa/configuring-two-factor-authentication) on your Github account).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove this section, as it refers to details on how to access a particular Jenkins server, versus what the features of the Jenkins pipeline code includes.

Comment on lines +9 to +11
## Grouping and Granularity
We have many tests, so it is important to be able to slice and dice them in different ways to be efficient with testing, debugging, and triage.
Tests are organized into groups typically based on where they came from and what type of test they are: `openjdk`, `perf`, `system`, `functional`, `external`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove this section as its answering "why" and is covered under other documentation versus being a reference on implementation details of how we run tests in a Jenkins environment.

Comment on lines +19 to +33
| Parameter | Description |
|--------------------------------------|-----------------------------------------------------------------------------------------------------------|
| `ADOPTOPENJDK_REPO` | Optional. Your fork of `aqa-tests`. |
| `ADOPTOPENJDK_BRANCH` | Optional. Your branch off of your fork of `aqa-tests`. |
| `OPENJ9_REPO` | Optional. Your fork of `openj9`. |
| `OPENJ9_BRANCH` | Optional. Your branch of your fork of `openj9`. |
| `OPENJ9_SHA` | Optional. Pin to a specific SHA of `openj9`. |
| `JDK_REPO` | Optional. Use test material from a specific OpenJDK repository. |
| `JDK_BRANCH` | Optional. Use test material from a specific OpenJDK branch. |
| `OPENJDK_SHA` | Optional. Pin to a specific OpenJDK SHA. |
| `TKG_OWNER_BRANCH` | Optional. Use a specific `adoptium/TKG` fork/branch. |
| `ADOPTOPENJDK_SYSTEMTEST_OWNER_BRANCH`| Optional. Use a specific `adoptium/aqa-systemtest` fork/branch. |
| `OPENJ9_SYSTEMTEST_OWNER_BRANCH` | Optional. Use a specific `openj9/openj9-systemtest` fork/branch. |
| `STF_OWNER_BRANCH` | Optional. Use a specific `adoptium/STF` fork/branch. |
| `JCK_GIT_REPO` | Optional. Use a specific private repository for JCK test material supplied under OCTLA. |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should add a column to the table to include the suggested default settings for parameters. Some of the descriptions need updating. Some parameters are not optional, but rather there are default values, so one does not need to fill in the value, because a default value is set.

doc/pages/JenkinFeatures.md Outdated Show resolved Hide resolved
Comment on lines +132 to +145
## Grinder Etiquette
- When possible, avoid running top-level targets (all of sanity.openjdk tests, etc., since we have nightly runs for those), or runs with 100x+ ITERATIONS.
These can take hours and will block the regular testing, especially on platforms where we have few machines.
- Do not mark Grinder runs as “keep forever,” there is a limited amount of space on Jenkins master.
Download any artifacts you need from your run as soon as it finishes to attach to issues, etc., and consider the Jenkins job transient.
- When reporting failures in issues, understand that Grinder jobs are transient (not kept for long on Jenkins).
Sharing a link to the Grinder is only useful for a short time.
Share the "Rerun in Grinder" link which has details of the impl/version/test target, output statistics & test artifacts in the issue; those are more concrete items when reproducing failures.
For openjdk test failures, here are some useful triage [instructions](https://github.com/adoptium/aqa-tests/wiki/Guidance-for-Creating-OpenJDK-Test-Defects).
- Proactively delete Grinder runs that you do not need (setup fails or mistake in target, etc.).
We keep the last 50 runs, and if you delete your unneeded runs immediately, other jobs will stay a little bit longer, giving others more time to grab artifacts.

**Note**: The above documentation provides a comprehensive reference for configuring Jenkins Test Jobs for AQAvit Grinder testing at ci.adoptium.net and OpenJ9 Jenkins server.
Make sure to follow the guidelines and etiquette to ensure efficient and effective testing processes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section can be removed from the reference documentation. It can go into a different doc that is not talking about the details of how the Jenkins pipeline function, but rather a request for how the community is expected to behave.

@smlambert
Copy link
Contributor

Thanks @Ndacyayisenga-droid - we will plan to merge this PR as is and make further updates on this documentation once its in.

@sophia-guo - can I get a review to get this in, and I will make the next phase of improvements shortly.

Copy link
Contributor

@smlambert smlambert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As noted, we will add this update as is, and continue to refine it in a follow-up PR.

@smlambert smlambert requested a review from sophia-guo September 7, 2023 14:34
@smlambert smlambert merged commit 90fa906 into adoptium:master Nov 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants