Skip to content

Commit

Permalink
fix(ci): Fix publishing of public docker images during releases of se…
Browse files Browse the repository at this point in the history
…rver pipelines (microsoft#22593)

## Description

The stage that publishes the public versions of docker images for the
server components is currently broken
([example](https://dev.azure.com/fluidframework/internal/_build/results?buildId=273242&view=logs&s=2ff3421f-f27a-5211-cd8c-afeb9905f8f6),
msft internal). The problem is that `$(containerTag)` is not getting
replaced with a real value. This PR adjusts the server pipelines to
address the issue.

I'm pretty confident the problem started [when we started using the 1ES
pipeline
templates](microsoft#19382).
Before that, [the build-docker-service.yml template had variables at the
root of the
file](https://github.com/seanimam/FluidFramework/blob/f14abf74bd4474275b0ba0553728e543c13e06f3/tools/pipelines/templates/build-docker-service.yml#L100),
so all stages in that file had access to them. When we moved to 1ES
pipeline templates, those variables moved to be specified in the build
stage and the publish stages lost access to them, which results in the
`$(containerTag)` variable that the publish stage is trying to use to
not getting replaced by ADO.

Changes in this PR:
- Remove the `include-publish-docker-service.yml` template (and CI/PR
triggers off it) which is pretty redundant, only importing another file
with no special logic.
- Introduces a new template `include-vars-docker.yml` that defines the
variables which are relevant to pipelines that produce docker images.
Almost a line-by-line copy of the variables that used to be in the build
stage of `build-docker-service.yml`, with a few adjustments:
- `tagNameOnReleaseImage` was removed from the list of variables and its
logic is now inlined (and simplified) in the `publish_docker_official`
stage of `build-docker-service.yml`.
- Removed the bit that set `latestContainerTag` in the case of the
`next` branch, since we don't have a `next` branch anymore.
- The new template is now used in both the build stage and the
`publish_docker_official` (the one that fails today). No other stages
seem to need the new variables.
- Fix a small typo `1 -> i`
  • Loading branch information
alexvy86 authored Sep 24, 2024
1 parent ff1b2c7 commit 64b64a0
Show file tree
Hide file tree
Showing 8 changed files with 82 additions and 80 deletions.
1 change: 0 additions & 1 deletion tools/pipelines/server-gitrest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ trigger:
- tools/pipelines/templates/include-publish-npm-package.yml
- tools/pipelines/templates/include-publish-npm-package-deployment.yml
- tools/pipelines/templates/include-publish-npm-package-steps.yml
- tools/pipelines/templates/include-publish-docker-service.yml
- tools/pipelines/templates/include-publish-docker-service-steps.yml
- tools/pipelines/templates/include-git-tag-steps.yml
- tools/pipelines/templates/include-use-node-version.yml
Expand Down
1 change: 0 additions & 1 deletion tools/pipelines/server-gitssh.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,6 @@ trigger:
- tools/pipelines/templates/build-docker-service.yml
- tools/pipelines/templates/include-vars.yml
- tools/pipelines/templates/include-install-pnpm.yml
- tools/pipelines/templates/include-publish-docker-service.yml
- tools/pipelines/templates/include-publish-docker-service-steps.yml
- tools/pipelines/templates/include-git-tag-steps.yml
- tools/pipelines/templates/include-use-node-version.yml
Expand Down
1 change: 0 additions & 1 deletion tools/pipelines/server-historian.yml
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,6 @@ trigger:
- tools/pipelines/templates/include-publish-npm-package.yml
- tools/pipelines/templates/include-publish-npm-package-deployment.yml
- tools/pipelines/templates/include-publish-npm-package-steps.yml
- tools/pipelines/templates/include-publish-docker-service.yml
- tools/pipelines/templates/include-publish-docker-service-steps.yml
- tools/pipelines/templates/include-git-tag-steps.yml
- tools/pipelines/templates/include-use-node-version.yml
Expand Down
1 change: 0 additions & 1 deletion tools/pipelines/server-routerlicious.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,6 @@ trigger:
- tools/pipelines/templates/include-publish-npm-package.yml
- tools/pipelines/templates/include-publish-npm-package-deployment.yml
- tools/pipelines/templates/include-publish-npm-package-steps.yml
- tools/pipelines/templates/include-publish-docker-service.yml
- tools/pipelines/templates/include-publish-docker-service-steps.yml
- tools/pipelines/templates/include-git-tag-steps.yml
- tools/pipelines/templates/include-use-node-version.yml
Expand Down
76 changes: 33 additions & 43 deletions tools/pipelines/templates/build-docker-service.yml
Original file line number Diff line number Diff line change
Expand Up @@ -156,41 +156,11 @@ extends:
displayName: Build Stage
dependsOn: [] # this stage doesn't depend on preceding stage
variables:
- ${{ if eq(parameters.testBuild, true) }}:
- name: buildContainerName
value: test/${{ parameters.containerName }}
- ${{ else }}:
- name: buildContainerName
value: build/${{ parameters.containerName }}
- name: containerTagSuffix
value: $[ format('0.0.{0}', variables['Build.BuildNumber']) ]
- name: baseContainerTag
value: base:$(containerTagSuffix)
- ${{ if eq(parameters.shouldPushDockerImage, false) }}:
- name: containerTag
value: $(buildContainerName):$(containerTagSuffix)
- ${{ else }}:
- name: containerTag
value: $(containerRegistryUrl)/$(buildContainerName):$(containerTagSuffix)
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- name: latestContainerTag
value: $(containerRegistryUrl)/$(buildContainerName):latest
- ${{ elseif eq(variables['Build.SourceBranch'], 'refs/heads/next') }}:
- name: latestContainerTag
value: $(containerRegistryUrl)/$(buildContainerName):next
- ${{ else }}:
# In this case we just want latestContainerTag defined so it gets replaced by ADO and Bash tasks don't
# complain about "command 'latestContainerTag' not found".
- name: latestContainerTag
value: ''
# tag the git repo when we published the docker image if the releaseKind is docker,
# otherwise tag with the publishing the packages (where the releaseKind is both or npm)
- ${{ if ne(parameters.releaseKind, 'docker') }}:
- name: tagNameOnReleaseImage
value:
- ${{ if eq(parameters.releaseKind, 'docker') }}:
- name: tagNameOnReleaseImage
value: ${{ parameters.tagName }}
- template: /tools/pipelines/templates/include-vars-docker.yml@self
parameters:
containerName: ${{ parameters.containerName }}
shouldPushDockerImage: ${{ parameters.shouldPushDockerImage }}
testBuild: ${{ parameters.testBuild }}
jobs:
- job: build
displayName: Build Container - ${{ parameters.containerName }}
Expand Down Expand Up @@ -565,7 +535,11 @@ extends:
publishLocation: 'pipeline'
sbomEnabled: false

# Publish stage
# Publish stages
# Note: normally (i.e. when the releaseKind is 'both') we tag the git repo when we publish the npm packages.
# This is also true if the releaseKind is 'npm' (meaning we only want to publish the npm packages).
# But if the releaseKind is 'docker' (meaning we only want to publish the docker images, not the npm packages),
# then those steps should tag the repo.
- ${{ if and(eq(parameters.shouldPublishNpmPackages, true), ne(parameters.releaseKind, 'docker')) }}:
- template: /tools/pipelines/templates/include-publish-npm-package.yml@self
parameters:
Expand All @@ -575,10 +549,26 @@ extends:
buildToolsVersionToInstall: ${{ parameters.buildToolsVersionToInstall }}

- ${{ if eq(parameters.shouldReleaseDockerImage, true) }}:
- template: /tools/pipelines/templates/include-publish-docker-service.yml@self
parameters:
containerRegistry: $(containerRegistryConnection)
containerTag: $(containerTag)
containerRegistryUrl: $(containerRegistryUrl)
containerName: ${{ parameters.containerName }}
tagName: $(tagNameOnReleaseImage)
- stage: publish_docker_official
dependsOn: build
displayName: Publish Official Docker Image
condition: and(succeeded(), eq(variables['release'], 'release'))
variables:
- template: /tools/pipelines/templates/include-vars-docker.yml@self
parameters:
containerName: ${{ parameters.containerName }}
shouldPushDockerImage: ${{ parameters.shouldPushDockerImage }}
testBuild: ${{ parameters.testBuild }}
jobs:
- template: /tools/pipelines/templates/include-publish-docker-service-steps.yml@self
parameters:
environment: container-registry-public
# $(containerRegistryConnection) and $(containerRegistryUrl) come from the container-registry-info variable
# group and need to be specified as runtime variables (variables from variable groups apparently are never
# available "statically" at parse/compile time, so can't be used with template-expression syntax ( '${{ }}' )).
containerRegistry: $(containerRegistryConnection)
containerRegistryUrl: $(containerRegistryUrl)
containerTag: $(containerTag)
containerName: public/${{ parameters.containerName }}
${{ if eq(parameters.releaseKind, 'docker') }}:
tagName: ${{ parameters.tagName }}
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
arguments: '${{ parameters.containerTag }} ${{ parameters.containerRegistryUrl }}/${{ parameters.containerName }}:$(containerTagSuffix)'

- task: Docker@1
displayName: Tag 1mage as 'latest'
displayName: Tag image as 'latest'
inputs:
command: 'tag'
arguments: '${{ parameters.containerTag }} ${{ parameters.containerRegistryUrl }}/${{ parameters.containerName }}:latest'
Expand Down
32 changes: 0 additions & 32 deletions tools/pipelines/templates/include-publish-docker-service.yml

This file was deleted.

48 changes: 48 additions & 0 deletions tools/pipelines/templates/include-vars-docker.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Copyright (c) Microsoft Corporation and contributors. All rights reserved.
# Licensed under the MIT License.

# common variables for pipelines that produce docker images

parameters:
# Name for the container image.
- name: containerName
type: string

# Whether the docker image should be pushed to our internal Azure Container Registry.
- name: shouldPushDockerImage
type: boolean
default: false

# If the build is running for a test branch
- name: testBuild
type: boolean
default: false

variables:
- name: buildContainerName
${{ if eq(parameters.testBuild, true) }}:
value: test/${{ parameters.containerName }}
${{ else }}:
value: build/${{ parameters.containerName }}
- name: containerTagSuffix
value: $[ format('0.0.{0}', variables['Build.BuildNumber']) ]
- name: baseContainerTag
value: base:$(containerTagSuffix)
# If docker images will be pushed, we need a variable where they are prefixed with the container registry URL.
# $(containerRegistryUrl) comes from the container-registry-info variable group and needs to be
# specified as a runtime variable (variables from variable groups apparently are never available "statically"
# at parse/compile time, so can't be used with template-expression syntax ( '${{ }}' )).
- ${{ if eq(parameters.shouldPushDockerImage, true) }}:
- name: containerTag
value: $(containerRegistryUrl)/$(buildContainerName):$(containerTagSuffix)
- ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/main') }}:
- name: latestContainerTag
value: $(containerRegistryUrl)/$(buildContainerName):latest
- ${{ else }}:
# In this case we just want latestContainerTag defined so it gets replaced by ADO and Bash tasks don't
# complain about "command 'latestContainerTag' not found".
- name: latestContainerTag
value: ''
- ${{ else }}:
- name: containerTag
value: $(buildContainerName):$(containerTagSuffix)

0 comments on commit 64b64a0

Please sign in to comment.