Job's "Started" notification missing at first build (multibranch and regular pipelines)

[mbxsuite]

Jenkins and plugins versions report

Environment

Jenkins: 2.375.2
OS: Linux - 5.4.0-137-generic
Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.18.0
active-directory:2.30
antisamy-markup-formatter:155.v795fb_8702324
apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61
authentication-tokens:1.4
authorize-project:1.4.0
basic-branch-build-strategies:71.vc1421f89888e
blueocean:1.27.1
blueocean-autofavorite:1.2.5
blueocean-bitbucket-pipeline:1.27.1
blueocean-commons:1.27.1
blueocean-config:1.27.1
blueocean-core-js:1.27.1
blueocean-dashboard:1.27.1
blueocean-display-url:2.4.1
blueocean-events:1.27.1
blueocean-git-pipeline:1.27.1
blueocean-github-pipeline:1.27.1
blueocean-i18n:1.27.1
blueocean-jwt:1.27.1
blueocean-personalization:1.27.1
blueocean-pipeline-api-impl:1.27.1
blueocean-pipeline-editor:1.27.1
blueocean-pipeline-scm-api:1.27.1
blueocean-rest:1.27.1
blueocean-rest-impl:1.27.1
blueocean-web:1.27.1
bootstrap4-api:4.6.0-5
bootstrap5-api:5.2.1-3
bouncycastle-api:2.27
branch-api:2.1071.v1a_188a_562481
caffeine-api:2.9.3-65.v6a_47d0f4d1fe
checks-api:1.8.1
cloudbees-bitbucket-branch-source:791.vb_eea_a_476405b
cloudbees-folder:6.800.v71307ca_b_986b
command-launcher:90.v669d7ccb_7c31
commons-httpclient3-api:3.1-3
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-27.vb_fa_3896786a_7
config-file-provider:3.11.1
credentials:1214.v1de940103927
credentials-binding:523.vd859a_4b_122e6
data-tables-api:1.12.1-4
dependency-track:4.2.0
display-url-api:2.3.7
docker-commons:1.21
docker-workflow:563.vd5d2e5c4007f
dtkit-api:3.0.2
durable-task:504.vb10d1ae5ba2f
echarts-api:5.4.0-1
email-ext:2.93
embeddable-build-status:312.vf2de01b_051d0
favorite:2.4.1
font-awesome-api:6.2.1-1
git:5.0.0
git-client:4.1.0
git-server:99.va_0826a_b_cdfa_d
github:1.36.1
github-api:1.303-400.v35c2d8258028
github-branch-source:1701.v00cc8184df93
handlebars:3.0.8
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
htmlpublisher:1.31
instance-identity:142.v04572ca_5b_265
ionicons-api:31.v4757b_6987003
jackson2-api:2.14.2-319.v37853346a_229
jakarta-activation-api:2.0.1-2
jakarta-mail-api:2.0.1-2
javax-activation-api:1.2.0-5
javax-mail-api:1.6.2-8
jaxb:2.3.7-1
jdk-tool:63.v62d2fd4b_4793
jenkins-design-language:1.27.1
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.6.1-2
jsch:0.1.55.61.va_e9ee26616e7
junit:1166.va_436e268e972
lockable-resources:1122.v14c3d52cb_1b_1
mailer:448.v5b_97805e3767
matrix-auth:3.1.6
matrix-project:785.v06b_7f47b_c631
mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a
mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a
multibranch-build-strategy-extension:1.0.10
okhttp-api:4.9.3-108.v0feda04578cf
pipeline-build-step:2.18
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:629.vb_5627b_ee2104
pipeline-input-step:466.v6d0a_5df34f81
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2118.v31fd5b_9944b_5
pipeline-model-definition:2.2118.v31fd5b_9944b_5
pipeline-model-extensions:2.2118.v31fd5b_9944b_5
pipeline-rest-api:2.31
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5
pipeline-stage-view:2.31
pipeline-utility-steps:2.15.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:2.20.0
popper-api:1.16.1-3
popper2-api:2.11.6-2
pubsub-light:1.17
resource-disposer:0.20
scm-api:631.v9143df5b_e4a_a
script-security:1229.v4880b_b_e905a_6
snakeyaml-api:1.33-90.v80dcb_3814d35
sonar:2.15
sse-gateway:1.26
ssh-agent:295.v9ca_a_1c7cc3a_a_
ssh-credentials:305.v8f4381501156
ssh-slaves:2.854.v7fd446b_337c9
ssh-steps:2.0.65.vd26b_5b_9b_de4d
sshd:3.275.v9e17c10f2571
structs:324.va_f5d6774f3a_d
timestamper:1.21
token-macro:321.vd7cc1f2a_52c8
trilead-api:2.84.v72119de229b_7
uno-choice:2.6.4
variant:59.vf075fe829ccb
workflow-api:1208.v0cc7c6e0da_9e
workflow-basic-steps:994.vd57e3ca_46d24
workflow-cps:3611.v201b_d9f9eb_f7
workflow-durable-task-step:1223.v7f1a_98a_8863e
workflow-job:1268.v6eb_e2ee1a_85a
workflow-multibranch:716.vc692a_e52371b_
workflow-scm-step:400.v6b_89a_1317c9a_
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.44
xunit:3.1.2

What Operating System are you using (both controller, and any agents involved in the problem)?

agent: ubuntu:20.04
controller: docker jenkins/jenkins:lts-jdk11

Reproduction steps

case 1, simple pipeline:

1. create simple pipeline job
2. run build # 1
3. modify job to include notification option as:

[[
        name: 'Office 365',
        url: 'https://xxxxx.webhook.office.com/webhookb2/xxxxx',
        startNotification: true,
        notifySuccess: true,
        notifyAborted: false,
        notifyNotBuilt: false,
        notifyUnstable: false,
        notifyFailure: true,
        notifyBackToNormal: true,
        notifyRepeatedFailure: false
]]

4. run build # 2
5. run build # 3

case 2: multibranch pipeline:

1. create a Jenkinsfile for a multibranch pipeline job
2. include notification option as:

[[
        name: 'Office 365',
        url: 'https://xxxxx.webhook.office.com/webhookb2/xxxxx',
        startNotification: true,
        notifySuccess: true,
        notifyAborted: false,
        notifyNotBuilt: false,
        notifyUnstable: false,
        notifyFailure: true,
        notifyBackToNormal: true,
        notifyRepeatedFailure: false
]]

3. push Jenkinsfile to Git / branch "main"
4. create a new branch "A" from "main"
5. run build # 1 for branch "A"
6. run build # 2 for branch "A"
7. create a new branch "B" from "main"
8. run build # 1 for branch "B"
9. run build # 2 for branch "B"

Expected Results

case 1, simple pipeline:

• build # 1 no notification
• build # 2
  • "started" notification received
  • "success" notification received
• build # 3
  • "started" notification received
  • "success" notification received

case 2: multibranch pipeline:

• branch "A"
  • build # 1
    • "started" notification received
    • "success" notification received
  • build # 2
    • "started" notification received
    • "success" notification received
• branch "B"
  • build # 1
    • "started" notification received
    • "success" notification received
  • build # 2
    • "started" notification received
    • "success" notification received

Actual Results

case 1, simple pipeline:

• build # 1 no notification
• build # 2
  • "success" notification received
• build # 3
  • "started" notification received
  • "success" notification received

case 2: multibranch pipeline:

• branch "A"
  • build # 1
    • "success" notification received
  • build # 2
    • "started" notification received
    • "success" notification received
• branch "B"
  • build # 1
    • "success" notification received
  • build # 2
    • "started" notification received
    • "success" notification received

Anything else?

Case 2 will be especially painful, when you run pipelines that deploy specifically named branches for product versions to critical target environments, like qa or production systems.

You simply want to know when a critical system is running a deployment. Hence, getting the right notifications to target audience.

example:
Dedicated multibranch pipeline job is deploying artifacts to remote systems. source is a "named" release branch of that job.

In this case it is very likely that the release branch has never been seen by the pipeline. Due to this bug, no one gets notified of that process starting.

[damianszczepanik]

I have cloned regular job, ran it and got the notification in MS Teams

[mbxsuite]

Did some more tests:

• The very first "started" notification is only received, when manually creating a pipeline job and additionally configuring the WebHook manually in the job configuration UI.
• The very first "started" notification is not received, when creating a pipelione job and let the pipeline script define the WebHook via the "options" section.

The Bug occurs for Pipelines via SCM (where the Jenkinsfile configures the job options, triggers) and in Multibranch-Pipeline Jobs for the same reason. After parsing the Jenkinsfile and adding the webhook for the very first time the "started" notification is neglected. If it was configured before (by a previous run or manually) it will be send out.

You may now argue the Job was already started,

• triggered Build
• read config
• read Jenkinsfile
• parsed Jenkinsfile
• add new web-hook options as defined in Jenkinsfile
• run rest of the pipeline
  which may technically correct.
  But this will still be an issue where the web-hook Options are not manually configured and rather read from scm/Jenkinsfile as in Multibranch Pipelines.

[damianszczepanik]

OK, so based on my experiences with Jenkinsfile and the way how this work when loading it from git repository this is not the plugin issue but the configuration of your job. I believe that for the first run you do not have configured hooks and that's why they are not executed.
Then, when job is running - not only first time but also third and fourth and so on - job configuration is updated based on what you have in your Jenkinsfile. This is by Jenkins design.

You can probably prove it by comparing config.xml version after the project is setup and after the first execution is completed. Look for options/properties section to see what has changed after the job execution.

Or similar: You have your hook configured and then you decided that they should be removed. Do this in git repository and check what happens for the next job run: will the notification be sent because you have it OR will not be sent because you have decided to remove it