Unable to re-request Github check via Github following upgrade from 1.0.16 to 1.0.18

[steele]

Jenkins and plugins versions report

Environment

Jenkins: 2.332.1
OS: Linux - 4.15.0-166-generic
---
ace-editor:1.1
active-directory:2.25.1
amazon-ecr:1.7
analysis-model-api:10.9.3
antisamy-markup-formatter:2.7
apache-httpcomponents-client-4-api:4.5.13-1.0
authentication-tokens:1.4
aws-credentials:191.vcb_f183ce58b_9
aws-java-sdk:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-cloudformation:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-codebuild:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ec2:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ecr:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ecs:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-elasticbeanstalk:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-iam:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-logs:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-minimal:1.12.163-315.v2b_716ec8e4df
aws-java-sdk-ssm:1.12.163-315.v2b_716ec8e4df
basic-branch-build-strategies:1.3.2
blueocean:1.25.3
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.25.3
blueocean-commons:1.25.3
blueocean-config:1.25.3
blueocean-core-js:1.25.3
blueocean-dashboard:1.25.3
blueocean-display-url:2.4.1
blueocean-events:1.25.3
blueocean-git-pipeline:1.25.3
blueocean-github-pipeline:1.25.3
blueocean-i18n:1.25.3
blueocean-jira:1.25.3
blueocean-jwt:1.25.3
blueocean-personalization:1.25.3
blueocean-pipeline-api-impl:1.25.3
blueocean-pipeline-editor:1.25.3
blueocean-pipeline-scm-api:1.25.3
blueocean-rest:1.25.3
blueocean-rest-impl:1.25.3
blueocean-web:1.25.3
bootstrap4-api:4.6.0-3
bootstrap5-api:5.1.3-6
bouncycastle-api:2.25
branch-api:2.7.0
caffeine-api:2.9.2-29.v717aac953ff3
checks-api:1.7.2
cloudbees-bitbucket-branch-source:757.vddedc5f2589a_
cloudbees-folder:6.708.ve61636eb_65a_5
cobertura:1.17
code-coverage-api:2.0.4
command-launcher:1.6
configuration-as-code:1414.v878271fc496f
copyartifact:1.46.3
credentials:1074.v60e6c29b_b_44b_
credentials-binding:1.27.1
dark-theme:156.v6cf16af6f9ef
data-tables-api:1.11.4-3
display-url-api:2.3.5
docker-commons:1.19
docker-workflow:1.28
durable-task:493.v195aefbb0ff2
ec2:1.68
echarts-api:5.3.0-2
email-ext:2.87
envinject-api:1.180.v98d833b_27470
favorite:2.4.1
file-operations:1.11
font-awesome-api:6.0.0-1
forensics-api:1.12.0
git:4.10.3
git-client:3.11.0
git-server:1.10
github:1.34.3
github-api:1.301-378.v9807bd746da5
github-branch-source:1583.v18d333ef7379
github-checks:1.0.18
github-pr-comment-build:61.v49f749d31d98
github-scm-trait-notification-context:1.1
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-1.0
htmlpublisher:1.29
http_request:1.14
jackson2-api:2.13.2-260.v43d711474c77
javadoc:217.v905b_86277a_2a_
javax-activation-api:1.2.0-2
javax-mail-api:1.6.2-5
jaxb:2.3.0.1
jdk-tool:1.5
jenkins-design-language:1.25.3
jersey2-api:2.35-4
jira:3.7
jira-steps:1.6.0
jjwt-api:0.11.2-9.c8b45b8bb173
job-dsl:1.78.3
jquery-detached:1.2.1
jquery3-api:3.6.0-2
jsch:0.1.55.2
junit:1.56
lockable-resources:2.14
mailer:408.vd726a_1130320
matrix-auth:3.1
matrix-project:758.v7a_ea_491852f3
maven-plugin:3.18
mercurial:2.16
momentjs:1.1.1
monitoring:1.90.0
multibranch-build-strategy-extension:1.0.10
node-iterator-api:1.5.1
okhttp-api:4.9.3-105.vb96869f8ac3a
permissive-script-security:0.7
pipeline-build-step:2.16
pipeline-github:2.8-138.d766e30bb08b
pipeline-githubnotify-step:49.vf37bf92d2bc8
pipeline-graph-analysis:188.v3a01e7973f2c
pipeline-input-step:446.vf27b_0b_83500e
pipeline-milestone-step:100.v60a_03cd446e1
pipeline-model-api:2.2064.v5eef7d0982b_e
pipeline-model-definition:2.2064.v5eef7d0982b_e
pipeline-model-extensions:2.2064.v5eef7d0982b_e
pipeline-rest-api:2.23
pipeline-stage-step:291.vf0a8a7aeeb50
pipeline-stage-tags-metadata:2.2064.v5eef7d0982b_e
pipeline-stage-view:2.23
pipeline-utility-steps:2.12.0
plain-credentials:1.8
plugin-util-api:2.14.0
popper-api:1.16.1-2
popper2-api:2.11.2-1
prism-api:1.26.0-2
pubsub-light:1.16
resource-disposer:0.17
s3:0.12.1
scm-api:595.vd5a_df5eb_0e39
scm-filter-branch-pr:0.5.1
script-security:1140.vf967fb_efa_55a_
skip-notifications-trait:1.0.5
snakeyaml-api:1.29.1
solarized-theme:0.1
sse-gateway:1.25
ssh-agent:1.24.1
ssh-credentials:1.19
sshd:3.1.0
structs:308.v852b473a2b8c
swarm:3.31
theme-manager:0.6
timestamper:1.17
token-macro:280.v97a_82642793c
trilead-api:1.0.13
urltrigger:1.02
variant:1.4
warnings-ng:9.11.1
workflow-aggregator:2.7
workflow-api:1143.v2d42f1e9dea_5
workflow-basic-steps:941.vdfe1b_a_132c64
workflow-cps:2660.vb_c0412dc4e6d
workflow-cps-global-lib:564.ve62a_4eb_b_e039
workflow-durable-task-step:1121.va_65b_d2701486
workflow-job:1174.vdcb_d054cf74a_
workflow-multibranch:711.vdfef37cda_816
workflow-scm-step:2.13
workflow-step-api:622.vb_8e7c15b_c95a_
workflow-support:815.vd60466279fc8
ws-cleanup:0.40
xtrigger-api:0.4

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller is running Ubuntu Linux 18.04.
Github is GitHub Enterprise Server 3.2.6.

Reproduction steps

1. Have an integration between Jenkins (checks plugin 1.0.16) and Github with checks running
2. Create a PR, where the check run has failed and been reported as such
3. Upgrade Jenkins (including checks plugin to 1.0.18)
4. From the above PR in Github, request that the failed check run is re-run

Expected Results

The check run is re-executed by Jenkins.

Actual Results

No check run is executed by Jenkins.

Anything else?

This appears to have been caused by this change: c686ef4

The "external_id" format used has changed from Jenkins "Job" to Jenkins "Run", in the case of "stale" external_id references in Github, older "Job" ID format is not accepted and generates the exception shown below:

INFO i.j.p.c.g.CheckRunGHEventSubscriber#onEvent: Received rerun request through GitHub checks API.
SEVERE o.j.p.g.e.GHEventsSubscriber$4#applyNullSafe: Subscriber io.jenkins.plugins.checks.github.CheckRunGHEventSubscriber failed to process SCMEvent{type=UPDATED, <<<REDACTED>>>} hook, skipping...
java.lang.IllegalArgumentException: Invalid id
at hudson.model.Run.fromExternalizableId(Run.java:2477)
at io.jenkins.plugins.util.JenkinsFacade.getBuild(JenkinsFacade.java:225)
at io.jenkins.plugins.checks.github.CheckRunGHEventSubscriber.scheduleRerun(CheckRunGHEventSubscriber.java:108)
at io.jenkins.plugins.checks.github.CheckRunGHEventSubscriber.onEvent(CheckRunGHEventSubscriber.java:97)
at org.jenkinsci.plugins.github.extension.GHEventsSubscriber$4.applyNullSafe(GHEventsSubscriber.java:241)
at org.jenkinsci.plugins.github.extension.GHEventsSubscriber$4.applyNullSafe(GHEventsSubscriber.java:237)
at org.jenkinsci.plugins.github.util.misc.NullSafeFunction.apply(NullSafeFunction.java:18)
at com.google.common.collect.Iterators$6.transform(Iterators.java:826)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:52)
at com.google.common.collect.Iterators.addAll(Iterators.java:367)
at com.google.common.collect.Lists.newArrayList(Lists.java:147)
at com.google.common.collect.Lists.newArrayList(Lists.java:133)
at org.jenkinsci.plugins.github.util.FluentIterableWrapper.toList(FluentIterableWrapper.java:148)
at com.cloudbees.jenkins.GitHubWebHook.doIndex(GitHubWebHook.java:124)
at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:398)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:410)
at org.jenkinsci.plugins.github.webhook.RequirePostWithGHHookPayload$Processor.invoke(RequirePostWithGHHookPayload.java:80)
at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:208)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:141)
at org.kohsuke.stapler.IndexDispatcher.dispatch(IndexDispatcher.java:28)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.MetaClass$9.dispatch(MetaClass.java:475)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at com.cloudbees.jenkins.GitHubWebHookCrumbExclusion.process(GitHubWebHookCrumbExclusion.java:29)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:128)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:136)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)

[timja]

Interesting, the workaround isn't hard though, push a new commit / re-run the build once via Jenkins?

Contribution welcomed though if someone wants to PR it.

Will close in a month or so if no fix as the stale jobs will naturally go away

[tslmy]

Facing the same issue.

I understand the effort constraints, but please consider this a serious problem, because this leads people to mistakenly believe that they have actually re-requested a Check Run.

How is it possible that people may be tricked to believe so?

Because every time "Re-run" is clicked, GitHub will show a blue banner that says,

You have successfully requested [check name here] be rerun.

and we can't expect all contributors to all repos using GitHub Checks Plugin to understand how Jenkins work with GitHub and realize that they have to distrust GitHub and check Jenkins for themselves.

Hence, regarding this statement,

the workaround isn't hard though, push a new commit / re-run the build once via Jenkins?

I would say this:

It's not about whether the workaround is hard or easy; it's about whether making people realize that they have to distrust a platform that they have trusted for a long while (in our case, GitHub) is hard or easy. In a way, this bug is getting serious because it undermines trust. It's about psychology; It's about thinking in non-Jenkins-savvy developers' shoes.

Thanks for your consideration.

[KalleOlaviNiemitalo]

When github-checks-plugin doesn't find the build that it should restart, it logs a warning:

github-checks-plugin/src/main/java/io/jenkins/plugins/checks/github/CheckRunGHEventSubscriber.java

Lines 130 to 131 in c98fc7b

	LOGGER.log(Level.WARNING, String.format("No build found for rerun request from repository: %s and id: %s",
	repository.getFullName(), checkRun.getCheckRun().getExternalId()).replaceAll("[\r\n]", ""));

Is there a response that Jenkins could return to GitHub in this situation, to prevent GitHub from claiming "You have successfully requested [check name here] be rerun"?
If that can be done, it seems a worthwhile change regardless of whether the problem was caused by a recent upgrade of github-checks-plugin.

GitHub check_run event documentation does not say anything about possible responses. Best practices for using webhooks advises that "Your server should respond with a 2XX response within 10 seconds of receiving a webhook delivery", which I suppose means that GitHub checks the HTTP status code at least.

[timja]

So I understand if this is the same issue, are you upgrading from the ~2.5 year old version to a more recent one now and you've got a builds that won't re-trigger because the previous check has the old ID?