From 3670ea3ad0b07d79dfc6b21de9add545e9297b08 Mon Sep 17 00:00:00 2001 From: Andriy Redko Date: Fri, 20 Oct 2023 13:40:46 -0400 Subject: [PATCH] Update Github pull request template to have a task for inspecting failing checks (#10751) (#10794) * Update Github pull request template to have a task for inspecting failing checks * Add failing builds troubleshooting section to CONTRIBUTING.md * Address review comments --------- (cherry picked from commit ffe9371b38f98a2c9314bb6cfed4f8f224ff085c) Signed-off-by: Andriy Redko Signed-off-by: github-actions[bot] Co-authored-by: github-actions[bot] --- .github/pull_request_template.md | 3 ++- CONTRIBUTING.md | 12 ++++++++++++ 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index d7981b5113972..637d013f7b661 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -9,7 +9,8 @@ - [ ] All tests pass - [ ] New functionality has been documented. - [ ] New functionality has javadoc added -- [ ] Commits are signed per the DCO using --signoff +- [ ] Failing checks are inspected and point to the corresponding known issue(s) (See: [Troubleshooting Failing Builds](../blob/main/CONTRIBUTING.md#troubleshooting-failing-builds)) +- [ ] Commits are signed per the DCO using --signoff By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 560d55914c764..731cc5b0ee474 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -14,6 +14,7 @@ OpenSearch is a community project that is built and maintained by people just li - [Contributing Code](#contributing-code) - [Developer Certificate of Origin](#developer-certificate-of-origin) - [Review Process](#review-process) + - [Troubleshooting Failing Builds](#troubleshooting-failing-builds) ## First Things First @@ -120,3 +121,14 @@ During the PR process, expect that there will be some back-and-forth. Please tr If we accept the PR, a [maintainer](MAINTAINERS.md) will merge your change and usually take care of backporting it to appropriate branches ourselves. If we reject the PR, we will close the pull request with a comment explaining why. This decision isn't always final: if you feel we have misunderstood your intended change or otherwise think that we should reconsider then please continue the conversation with a comment on the PR and we'll do our best to address any further points you raise. + +## Troubleshooting Failing Builds + +The OpenSearch testing framework offers many capabilities but exhibits significant complexity (it does lot of randomization internally to cover as many edge cases and variations as possible). Unfortunately, this posses a challenge by making it harder to discover important issues/bugs in straightforward way and may lead to so called flaky tests - the tests which flip randomly from success to failure without any code changes. + +If your pull request reports a failing test(s) on one of the checks, please: + - look if there is an existing [issue](https://github.com/opensearch-project/OpenSearch/issues) reported for the test in question + - if not, please make sure this is not caused by your changes, run the failing test(s) locally for some time + - if you are sure the failure is not related, please open a new [bug](https://github.com/opensearch-project/OpenSearch/issues/new?assignees=&labels=bug%2C+untriaged&projects=&template=bug_template.md&title=%5BBUG%5D) with `flaky-test` label + - add a comment referencing the issue(s) or bug report(s) to your pull request explaining the failing build(s) + - as a bonus point, try to contribute by fixing the flaky test(s)