Merge queue not re-embarking PR after failed CI builds are rerun and succeeding

[ssnielsen]

Like many others, I assume, we require our CI pipeline (we use Circle CI) to successfully report back as succeeded before allowing Mergify to do its thing.

CI pipelines might fail for a number of reasons (flaky tests, missing network resources, etc.) that doesn't necessarily have anything to do with one's changes, which is sometimes fixable by a re-run of the CI job. I.e. one or more jobs goes from "failed" to "success".

Github does however see that the previously failed CI job is now succeeding which makes it a bit hard to spot why Mergify thinks it cannot continue.
One has to go to the /checks page to see the details (example: https://github.com/my-org/my-repo/pull/10095/checks)

I have the feeling that we've missed a configuration change - but I've gone through Mergify's changelogs + docs, with no glaring hints at what could be wrong, so any help is really appreciated.

Expected Behavior

In the past (up until around a month ago), I feel Mergify has automatically picked these things up and has re-added the PRs on which this pattern happened to the merge queue.

Actual Behavior

Since around a month ago, we've had to manually run the @mergifyio refresh command when the re-run CI builds succeeded to make Mergify aware of the now succeeding CI pipeline.

Steps to Reproduce the Problem

1. Make PR that results in a failing build
2. Fix the issue by re-running the failed CI jobs
3. Realize that Mergify doesn't see the previously failed jobs as succeeded - Github does however

Specifications

The repository is private, and I wanted to hear whether anybody else had experienced this before spending too much time on a publicly reproducing repo. So only left the relevant bits of the mergify config directly in here in the issue.

• Pull Request URL:
• Mergify Config URL:

queue_rules:
 - name: merge-queue
   conditions:
     - "check-success=build_and_test"

pull_request_rules:
  - name: Automatic merge
    conditions:
      # To be merged automatically, a PR must be based off master
      - "base=master"
      # ... and have at least 1 approval
      - "#approved-reviews-by>=1"
      # ... and all reviewers have reviewed
      - "#review-requested=0"
      # ... and all requests must be addressed
      - "#changes-requested-reviews-by=0"
      # ... and the "build_and_test" CI workflow must have completed successfully
      - "check-success=build_and_test"
      # ... and have the "Ready to merge" label applied. This lets us control when we want to queue the PR for merge
      - "label=Ready to merge"
    actions:
      queue:
        name: merge-queue

[jd]

Don't spend time reproducing the issue: if you spot this issue, just send a request to support with the PR URL and we'll look into it.

[ssnielsen]

Don't spend time reproducing the issue: if you spot this issue, just send a request to support with the PR URL and we'll look into it.

I spoke to a colleague, that said the same thing - already done 😉