ci.jenkins.io incorrectly shows build dates in Dec 1969 (the epoch) #3934
Comments
|
Oh that explains a lot! I did not pay enough attention but this is the (indirect) root cause of #3933. For info, it happens after a set of plugins upgrades on ci.jenkins.io + VM restart last sunday (4th). Note that ci.jenkins.io did break during the restart due to a Datadog stack overflow error (related to https://github.com/jenkinsci/datadog-plugin/blob/master/CHANGELOG.md#600--2024-01-31) 
=> this could explain the corruption but I don't have any real proof. |
|
As a matter of safety, let's take a snapshot of all our controllers Jenkins home before proceeding with further upgrades (ref. jenkins-infra/kubernetes-management#4941 and jenkins-infra/kubernetes-management#4942) |
Done for all controllers |
|
https://issues.jenkins.io/browse/JENKINS-66328 You also have the same (probably unrelated) additional problem of the labels incorrectly showing a daylight savings time. |
|
Another stackoverflow today, corresponding stack trace: https://gist.github.com/lemeurherve/eaf08053a133733a829c24ca0eec7117 Resolved by a docker restart. Reported on jenkinsci/datadog-plugin#389 too. |
Recover from job definition damage Avoid stack trace and page creation failure jenkins-infra/helpdesk#3934 https://gist.github.com/lemeurherve/eaf08053a133733a829c24ca0eec7117
|
The incorrect dates display is likely related to jenkinsci/datadog-plugin#393 in Jenkins Datadog plugin. It should be resolved in the plugin v6.0.1. |
|
Hello @nikita-tkachenko-datadog, I've responded in your issue at jenkinsci/datadog-plugin#393 (comment), we still have previous builds appearing as failed in 1970 on ci.jenkins.io even with the 6.0.1 version. Thanks for the quick stack overflow fix! |
|
https://issues.jenkins.io/browse/JENKINS-66328 describes a similar issue. So while there is a plugin data deserialisation problem in Datadog v6.0.0, it is possible that the date/status display issue is caused by something else. I apologise if my previous comment caused confusion. |
No worries, at least we are all in sync. Our goal was to share the knowledge so we can all build a better Jenkins ecosystem! |
FYI, another stack overflow on ci.jenkins.io while restarting the controller for a Docker-CE upgrade. A restart did the trick with no further corruption (cc @smerle33 for info) |
Indeed. I have never had the Datadog plugin installed, and I have pinned the problem as starting for me at some point between the 5th and the 19th of January. Further, downgrading Jenkins core fixes it. |
Can you provide the detailed plugin lists in the "before" and "after" state? I suspect it is more likely related to a plugin update than to anything that changed in Jenkins core. |
I already provided it on the linked issue.
Then why is it an update to Jenkins core that causes it? |
|
This listing can be used to narrow down the plugins that may have been upgraded before the problem started. As detailed on the linked issue, I already investigated |
I assume that it is not an update to Jenkins core that causes it. As far as I understand it, you updated Jenkins core and Jenkins plugins at the same time. The bad behavior on ci.jenkins.io seems to be due to the datadog plugin update. I assume in your environment there is some other plugin update that is causing unexpected behavior. |
|
If I may, can we split subjects? This issue tracker is not aimed at solving problems for user-related Jenkins installations, but only for the @OrangeDog you are correct, in your case it is something else than the datadog plugin that caused the build history corruption. As underlined by Nikita in jenkinsci/datadog-plugin#393 (comment), there might be a weird behavior in the Core itself during the recovery process which corrupts part of the build records. But it can be caused by many things including a plugin upgrade and / or resource issues (hard restart during But it has to be tracked in the JIRA issue with mentions to Nikita's comment (if not already the case): there is nothing that the Jenkins infrastructure team can do here to help, except reporting and raising the concern to the community. |
That sounds good to me. This issue is tracking the problem as encountered on ci.jenkins.io. JENKINS-66328 is tracking the report from @OrangeDog and others. |
|
@MarkEWaite except it is fixed by downgrading core, not by downgrading any plugins. Edit: it's not, it just made it much harder to reproduce |
|
Going to close this issue as we've secured the datadog plugin: it is using the 6.0.2 version everywhere:
The ci.jenkins.io builds are going to be cleaned up over time as we have global build discarding. |
Service(s)
ci.jenkins.io
Summary
Build dates for the ci.jenkins.io packaging test job shows Dec 1969 (the epoch in my time zone)
Reproduction steps
I've seen recent reports of date related errors on Jenkins controllers, but could not find the references to link. I assumed that the date related error reports were associated with incorrect operating system date settings on the controller, but I'm reasonably confident that the issue on ci.jenkins.io is not related to incorrect operating system date.
Reports of a similar issue include:
The text was updated successfully, but these errors were encountered: