[Update Center] Add archives.jenkins.io as a mirror fallback for JSON metadatas

[dduportal]

#2649 introduced a new Update Center system built on top of mirrorbits.

We want to add the VM archives.jenkins.io as a "mirror fallback" for the Update Center instead of using the US East Cloudflare R2 mirror.

Rationale:

• We have to be resilient against a Cloudflare outage => a non Cloudflare mirror is required
• We need a non-S3 mirror => an HTTP + Rsync service is needed
• HA is not required => a single VM is 👍
• This service is only used when there is a mismatch of all mirrors with the mirrorbits reference + during Cloudflare outages: We need a service where outbound bandwidth is cheap => DigitalOcean is a good candidate
• We want to avoid heavy maintenance as much as possible => archive.jenkins.io DigitalOcean VM is already a download mirror with everything required (webservice, rsync, etc.)

Todo list:

• Check the network routes to archives.jenkins. Fix the security groups and routing if failing (+ updatecli manifest to track IPs if needed):
  • From the trusted agent-1 with rsync (update_center2)
  • From any trusted.ci ephemeral VM agent with rsync (crawler)
  • From mirrorbits services inside publick8s with both rsync and HTTP
• Set up an rsync credential to allow both trusted.ci to rsync from its jobs and mirrorbits to scan the service with rsync
  • Add credential in sops (private key) + kubernetes-management (from sops to mirrorbits pods)
  • Add credential in the trusted.ci ZIP definition (in sops) and update it in trusted.ci
• Set up update_center2 to also rsync to archives.jio (with the updated ZIP credential above)
• Set up crawler to also rsync to archives.jio (with the updated ZIP credential above)
• Once archives.jio is kept up to date, set up mirrorbis instances (both secured and unsecured) to use archives.jio as fallback
• Check metrics to monitor in DigitalOcean to track outbound bandwidth

[dduportal]

Pairing session with @smerle33 :

• archives.jenkins.io has a partial content of the Update Center

  • Populated by the mirrorsync script which pulls data from the OSUOSL mirror to archives.jio
  • We see that this "partial" subset is also available (sometimes given the update frequencies of mirrors) in https://get.jenkins.io/updates/dynamic-2.459/update-center.json?mirrorlist which duplicate (somehow) our new Update Center system
  • This partial subset is populated by the sync.sh script in the pkg machine which runs every 30 min
    • Get a partial subset of the UC data (in /var/www/updates.jenkins.io) and copy it to the /srv/release/jenkins reference base directory - https://github.com/jenkins-infra/jenkins-infra/blob/c9cce943f1d6638d45539d7b7819d9673d3b9d84/dist/profile/templates/mirror-scripts/sync.sh.erb#L35-L47
      • Note: it copies only the tools ./updates/ directory and directories containing a update-center.json file. But NOT the files residing in the /var/www/updates.jenkins.io/ base directory
    • Then, copies this UC subset from /srv/release/jenkins to the remote OSUOSL server - https://github.com/jenkins-infra/jenkins-infra/blob/c9cce943f1d6638d45539d7b7819d9673d3b9d84/dist/profile/templates/mirror-scripts/sync.sh.erb#L50

• This regular update of data in archives.jenkins.io is concurrent with our plan to update the same data (to use it as fallback) and could create accidental downgrades. We need to either stop doing the current "sync" in archives, or use another distinct directory.

=> Proposal: Let's start from an empty clean state and use a new path https://archives.jenkins.io/update-center/

• Avoid keeping legacy stuff (which will be decommissioned little by little)
• Avoid conflicts
• Avoid confusion (archives.jenkins.io/updates/updates/ is really confusing...).

[dduportal]

Update:

• We need to open SSH access from the trusted.ci ephemeral agents (permanent agent has routing rules not blocking) => @smerle33 (as we recently restricted outbound SSH to only GitHub globally)
• Mirrorbits pods should use a rsync server (no ssh client, no need for auth. easier to manage) installed in archives.jio => @dduportal
• New credential for archives.jio in SSH => @dduportal for the Puppet retrieval part, and pair with @smerle33 for updating the generation script
• Update center2/crawler new Rsync target to add => @smerle33

[dduportal]

Update:

• The ephemeral agents of trusted.ci are now able to reach archives.jenkins.io thanks to fix(trusted.ci): allow crawler to publish to pkg VM (and allow to publish to archives) azure#896
• publick8s also has network access to archives.jenkins.io

[dduportal]

Update: https://archives.jenkins.io/update-center/ can be used as an update center (tested on ci.jio) and is updated by the UC2 build.

[dduportal]

• Archives has been added as a fallback in feat(publick8s/UC) set archives.jio as mirrors fallback kubernetes-management#6015:
• We can track the network metrics with:
  • Datadog host dashabord for archives.do.jenkins.io VM: https://jenkins.datadoghq.com/dash/host/14508244600?fromUser=true&refresh_mode=sliding&from_ts=1733907693205&to_ts=1734080493205&live=true
  • DigitalOcean VM metrics has an outbound bandwidth metrics tracking (but in Mb/s so instant): https://cloud.digitalocean.com/droplets/376458021/graphs?i=e3795e&period=week
  • DigitalOcean also provides, in the billing section, a summary of total bandwidth. We have consumed almost 4Gb this month, which forecasts around 10 Gb. We don't see (yet?) a real increase due to adding it as UC fallback). we'll watch carefully, but since we have 5 Gb for free per month, and we pay around $25 monthly of additional, it is absolutely ok!