Skip to content

Latest commit

 

History

History
139 lines (96 loc) · 11.1 KB

README.md

File metadata and controls

139 lines (96 loc) · 11.1 KB
Raw

PR Testers

openjdk-build is an open source project, therefore, we need to ensure the the code that is being deployed to our master branch doesn't break any existing code and actually works as expected. To achieve this level of testing, we use various jobs to compile, lint and test the code as well as running demo pipelines in a controlled sandbox environment if needs be. The demo pipelines are colloquially known as "The PR Tester" where the others are generally just referred to as "checkname check".

When they're used

Except for the #openjdk-build-pr-tester, all of the test groups are executed automatically on every PR and are defined inside the .github/workflows directory. These tests lint & compile the code you have altered, as well as executing full JDK builds using your code. Every new pull request to this repository that alters any groovy code OR that will likely affect our Jenkins builds should have the PR tester (#openjdk-build-pr-tester) run on it at least once to verify the changes don't break anything significant inside a Jenkins build environment (documentation changes being excluded from this rule).

What they are

There are four "groups" of tests that can be run on each PR:

The results of these jobs will appear as GitHub Status Check Results at the bottom of the PR being tested: Image of PR Tester Checks

Compile

This group consists of GitHub Status Checks run inside GitHub itself. They compile and unit test any code changes you have made.

Groovy

  • A relatively small job, this runs our groovy compiler, checking over the repository to ensure any new code successfully runs and conforms to our test suite.
  • Should you encounter a compile error that is caused by a missing function or error similar to the one below, then you should update our test doubles & stubs that we use to run these tests and emulate a live jenkins environment (link to example).
    prTester/pr_test_pipeline.groovy: 99: [Static type checking] - Cannot find matching method PullRequestTestPipeline#downstreamCommitStatus(groovy.lang.Closure). Please check if the declared type is correct and if the method exists.
     @ line 99, column 33.
                                       downstreamCommitStatus {
VersionParsingTest > parsesJava11NightlyString() STANDARD_OUT
    =JAVA VERSION OUTPUT=
    openjdk version "11.0.3" 2019-04-16
    OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.3+9-201903122221)
    OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.3+9-201903122221, mixed mode)
    =/JAVA VERSION OUTPUT=
    matched
    11.0.3+9-201903122221

Linter

This group consists of GitHub Status Checks run inside GitHub itself. They lint / analyse any changes you make to ensure they conform to our writing standards.

Shellcheck

  • This job downloads and runs the Shellcheck script analysis tool in order to lint and compile any changes you have made to our bash scripts. It does this via the shellcheck.sh script.
  • The job will fail and inform the user in the log if there are any violations of our bash scripting standards. If you feel that some of the standards are too strict or irrelevant to your changes, please raise it in Slack:#testing.

Build

This group is a matrix of GitHub Status Checks run inside GitHub itself. They execute a full set of builds to various specifications, mimicking a user running a build locally

  • The group collects a varied mix of java versions, operating systems and VM variants that each execute build-farm/make-adopt-build-farm.sh, essentially running a full JDK build as if we were setting up and testing a new Jenkins machine OR as if it was running a build locally on your machine.
  • Each job is run inside a Docker container to ensure reliability between each build. For example, Linux builds use our centos7_build_image Docker container.
  • Due to GitHub ratelimiting how many status checks can be run in the space of a few minutes, these checks may take a little while to complete while they're stuck in the queue. Be patient however, as some of your changes may affect one build completely differently to another build.

openjdk-build-pr-tester

Seen in the PR Status Checks as pipeline-build-check, the job is located here

This job runs a set of sandbox pipelines to test the changes that you have made to our codebase in a mock live Jenkins environment. It's executed by a custom groovy script from the job itself:

node("master") {
    //String ghprbPullId = "953"
    //String branch = "pull/${ghprbPullId}"

    //String ghprbActualCommit = "e309f0c8c10df9515770d5d8ec37daeddbbe7a15"
    String branch = "${ghprbActualCommit}"
    String url = 'https://github.com/AdoptOpenJDK/openjdk-build'

    checkout([$class: 'GitSCM', branches: [[name: branch]], userRemoteConfigs: [[refspec: " +refs/pull/*/head:refs/remotes/origin/pr/*/head +refs/heads/master:refs/remotes/origin/master +refs/heads/*:refs/remotes/origin/*",
    url: url]]])

    Closure prTest = load "pipelines/build/prTester/pr_test_pipeline.groovy"

    prTest(
        branch,
        currentBuild,
        this,
        url
        ).runTests()
}
  • That groovy script executes our pr_test_pipeline which is the main base file for this job.
  • NOTE: This tester is only really worth running if your code changes affect our groovy code OR Jenkins environment. Otherwise, the #Build jobs are sufficient enough to flag any problems with your code.

Usage

The tester has it's own admin and white lists on Jenkins. If you are on either list, the PR tester will run against your PR whenever you comment #run tests and will also allow you access to various commands you can run on your own PR or on someone else's:

run tests

  • Executes a new #openjdk-build-pr-tester job against this PR. These jobs will populate the GitHub status checks field as described above. Please be patient as the tester does not run concurrently so it may take some time to execute the jobs if there is a long job queue. You can track the progress of it in Jenkins OR look at the status check message:

    • Example of a PR that is in the queue: Image of queued tester
    • One that is at the front of the queue and currently being tested: Image of building tester

  • When the tester is done, it will return a response comment to the PR with feedback on the testing similar to the following: Image of test result

  • The message will vary depending on the result of the test. Please remember however, that failed tests may be due to existing problems in the nightly builds, not your code. If you're unsure if the tests failed because of your changes or not, check our issue board and our triage doc for the existing error. If your job was aborted, check the log to see who aborted it.

    • 🟢SUCCESS 🟢 All the downstream jobs passed, congratulations!
    • 🟠FAILURE 🟠 Some of the downstream jobs failed OR the job was aborted. Check the link in the field at the bottom of the PR for the job link to see exactly where it went wrong.
    • 🔴ERROR 🔴 Something more serious went wrong with the tester itself. Please raise an issue with a link to the job, the error encountered and your PR that caused it (again, you can use the link at the bottom to see exactly what happened).
add to whitelist