more frequent Dockerfile builds

[glyg]

I hope it is OK to ask this here, and more appropriate than on the image.sc post

Is there a reason not to RUN yum update -y right after the CentOS image was pulled?

Best

Guillaume

[sbesson]

@glyg thanks for raising this issue. The forum post was equally as appropriate and is it on our radar to follow-up but the team is operating in reduced capacity with several of us attending ELMI this week.

Speaking from the IDR experience, we are using rolling deployments with the base packages being systematically upgraded after provisioning the VMs. So running yum update -y when building the Docker images would mirror this behavior and feels reasonable and fairly low-cost.
The primary downside from my perspective is whether this communicates the user expectation that the latest Docker image will always be up-to-date with the latest upstream packages. In that case, this is a necessary first step byt additional infrastructure will be required to rebuild the image periodically as discussed in the related post.

I'll leave @joshmoore to comment as he was more heavily involved in the initial work on these Docker images and there might be some rationale I am overlooking.

[glyg]

The forum post was equally as appropriate and is it on our radar to follow-up but the team is operating in reduced capacity with several of us attending ELMI this week.

Sure, I was not meaning to sound impatient sorry!

[glyg]

The primary downside from my perspective is whether this communicates the user expectation that the latest Docker image will always be up-to-date with the latest upstream packages. In that case, this is a necessary first step byt additional infrastructure will be required to rebuild the image periodically as discussed in the related post

Yes, I think we are going to try that on our side for the moment anyway, but I wonder about good tests to run when re-deploying so frequently

[sbesson]

but I wonder about good tests to run when re-deploying so frequently

At least on the OME side, the latest versions of these Docker images should be tested weekly via the scheduled GitHub Actions CI builds in:

• the Docker deployment example under https://github.com/ome/omero-deployment-examples
• the integration tests for most plugins under https://github.com/ome/omero-plugins using https://github.com/ome/omero-test-infra

[glyg]

Thanks @sbesson If I understand correctly, integration tests as ran in omero-test-infra but with our own docker-compose and set of plugins would be a first step.

[joshmoore]

Probably the biggest downside of running apt/yum update that I can think of is the size of the image. It means our "base" image is a custom centos rather than the upstream one that others may have already pulled.

[sbesson]

Ah forgot about the image size cost. A quick test certainly shows a 2.5 fold increase

(base) [sbesson@idr2-slot3 ~]$ docker images centos
REPOSITORY   TAG       IMAGE ID       CREATED        SIZE
centos       7         eeb6ee3f44bd   8 months ago   204MB
(base) [sbesson@idr2-slot3 ~]$ docker build -t centos_updated .
Sending build context to Docker daemon  101.1MB
Step 1/2 : FROM centos:7
 ---> eeb6ee3f44bd
Step 2/2 : RUN yum update -y
 ---> Running in abedb5d76066
...
Complete!
Removing intermediate container abedb5d76066
 ---> c9cbc8961302
Successfully built c9cbc8961302
Successfully tagged centos_updated:latest
(base) [sbesson@idr2-slot3 ~]$ docker images centos_updated
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
centos_updated   latest    c9cbc8961302   2 minutes ago   509MB

[glyg]

Thanks, this indeed is an issue. It is said here that they perform rolling updates monthly or "as needed for emergency fixes". Let's see whether my security conscious sysadmin finds this acceptable :)

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/omero-docker-images-security-upgrades/67815/6

[JulianHn]

While the size increase of course cannot be mitigated completely, it can at least be somewhat limited by performing a clean of all caches in the same Run command:

FROM centos:centos7
RUN yum update -y && yum -y clean all && rm -fr /var/cache

docker build -t centos-update-clean .
Cleaning up list of fastest mirrors
Removing intermediate container 755787065bb7
 ---> 25fe9406d9ef
Successfully built 25fe9406d9ef
Successfully tagged centos-update-clean:latest
❯ docker images centos-update-clean
REPOSITORY            TAG       IMAGE ID       CREATED         SIZE
centos-update-clean   latest    25fe9406d9ef   9 seconds ago   364MB

This should be closer to the actual amount of additional installed packages.

// Julian

[JulianHn]

Actually, concerning the sizes: Independent of what the decision on yum update is, there is opportunity to decrease the docker sizes by amending the RUN commands with a clean of the caches:

Dockerfile diff:

--- a/Dockerfile
+++ b/Dockerfile
@@ -8,9 +8,13 @@ ADD playbook.yml requirements.yml /opt/setup/
 
 RUN yum -y install epel-release \
     && yum -y install ansible sudo \
-    && ansible-galaxy install -p /opt/setup/roles -r requirements.yml
+    && ansible-galaxy install -p /opt/setup/roles -r requirements.yml \
+    && yum -y clean all \
+    && rm -fr /var/cache
 
-RUN ansible-playbook playbook.yml
+RUN ansible-playbook playbook.yml \
+    && yum -y clean all \
+    && rm -fr /var/cache

I get the following image sizes (Clean & Update has an additional yum update -y in the first RUN command):

Options	Image Size
Current	1.29GB
Clean	918MB
Clean & Update	1.05GB

So this might be something to think about.

// Julian

[joshmoore]

If we had an auto-redeploy (i.e. bumping the final -1 build number) every time that the upstream centos image is released, would there still be a desire to keep yum update in the build? If not, I'd be more in favor of more frequent rebuilds. (Either via a new issue or by updating this one)

[JulianHn]

Looking at the CentOS tags , it seems like the base repo is not updated very frequently. Last tag was pushed 4 monts ago. I assume that security updates to yum packages might happen more frequently.

A more general question: Does it make sense to put much more work into the CentOS image? I assume that this Docker will probably switch over to a different distribution in the future (Alma ?), since CentOS changed its distribution model?

// Julian

[JulianHn]

Looking at the CentOS tags , it seems like the base repo is not updated very frequently. Last tag was pushed 4 monts ago. I assume that security updates to yum packages might happen more frequently.

Actually, docker run -it centos:centos7 yum updateinfo list security all returns an empty list, so apparently I'm mistaken.
I agree then with @joshmoore that auto-triggering a rebuild upon finding a new digest on the centos:centos7 would be the nicest way to go forward. I'm unfortunately not very familiar with Github Workflows (we are using a self-hosted GitLab instance at the institute), so I can't comment on how long it would take to achieve this.

// Julian

[glyg]

Even this is empty: docker run -it centos:centos7 yum updateinfo list all
So yay for more frequent builds — ok to rename the issue accordingly

[joshmoore]

I've opened ome/omero-server-docker#67 as a first attempt.

[glyg]

Hi again,
Although the centos 7 image was built 4 months ago according to docker hub
and indeed the yum updateinfo list security all is empty for this image, it is not the case for the latest omero-server image that was built 20 days ago, I checked I had the correct image — same sha256 digest, see the attached list:

sec_list.txt

Maybe the image needs to be built with --no-cache?

Best,

G.

[joshmoore]

Odd. I wouldn't expect that --no-cache is needed since https://github.com/ome/omero-server-docker/blob/master/.github/workflows/main.yml doesn't have any caching configured.

[manics]

Is that listing all available security releases instead of just updates? It includes things like qemu.

[glyg]

So yum updateinfo security all lists all the escurity issues, more relevant here is yum updateinfo security installed for the actually installed packages

Here, what we get is only one "Low" security with python2-httplib2

docker run -d --name omero openmicroscopy/omero-server:5 \
  && docker exec -u root omero yum updateinfo security installed \
  && docker container stop omero && docker container rm omero 

outputs

...
FEDORA-EPEL-2020-f33a36b2c4 Low/Sec. python2-httplib2-0.18.1-3.el7.noarch
...

So this is not so bad, is it?

I also tried docker scan with docker scan openmicroscopy/omero-server:5 which spits out a lot of stuff. I'm not sure how to account for the result, as it looks also into the java components etc. I'm attaching the output anyway.
docker_scan_omero-server.txt

Best
Guillaume

PS:

This (with updates instead of installed):

docker run -d --name omero openmicroscopy/omero-server:5 && docker exec -u root omero yum updateinfo security updates && docker container stop omero && docker container rm omero

Is empty, so yum update won't fix this security issue

[JulianHn]

Hey all,

If you look at the packages with securityinfo, they are all epel packages. These are not included in the centos base image, but installed in the omero-server Dockerfile.

https://github.com/ome/omero-server-docker/blob/23b0c8cda5288c2b2d60022b5593853cac7bd7dc/Dockerfile#L11

AFAICS this would only be mitigated by rebuilding OMERO.

// Julian

[joshmoore]

Ah, interesting. Thanks, @JulianHn. Off-hand I don't see an official epel-enabled base image though there are a few on Docker Hub. One option would be to host our own and detect when a rebuild is needed via a cron job. Thoughts?

[JulianHn]

If dont see how this would work. The base Image would not know which packages would br installed by the omero containers. Why not let the cron Job run on the omero images and trigger a rebuild when one of the installed packages has a security info and an actual update ready? E.g. At the moment no rebuild is necessary because the security issue is not patched anyway.

[joshmoore]

The base Image would not know which packages would br installed by the omero containers.

You're right. We would need to define a base image that has all the packages we need.

At the moment no rebuild is necessary because the security issue is not patched anyway.

👍

[JulianHn]

So ... I was trying to come up with a github action to automate the checks and noticed that even with CentOS images from 2014, no securityinfo popped up for the base repository. It turns out, that CentOS does not tag their repository with metadata, therefore this will not work (cf. https://forums.centos.org/viewtopic.php?t=65588).
It would be possible to check the EPEL packages, but for the base packages I think the only option is to trust the CentOS pipelines to rebuild their Docker image, whenever there is a security update.

[JulianHn]

Okay a quick further check: It seems it is not possible to rely on CentOS.
An easy example

$ docker run --rm -it centos:centos7
$ yum check-update
    [...]
    zlib.x86_64                       1.2.7-20.el7_9         updates
$ yum list zlib
    [...]
    Installed Packages
    zlib.x86_64                       1.2.7-18.el7           @CentOS
    Available Packages
    zlib.i686                         1.2.7-20.el7_9         updates
$ yum update -y zlib && yum install -y  changelog && yum changelog zlib | head -15
    [...]
    ==================== Installed Packages ====================
    zlib-1.2.7-20.el7_9.x86_64               installed
    * Tue Apr 12 12:00:00 2022 Matej Mu?ila <[email protected]> - 1.2.7-20
    - Resolves: CVE-2018-25032

That CVE-2018-25032, has a score of 8.2 so high severity and is not fixed in the base images :(

[joshmoore]

🤯 https://snyk.io/test/docker/centos%3A7 certainly shows the same (e.g. https://security.snyk.io/vuln/SNYK-CENTOS7-ZLIB-2434395)

That would seem to suggest we need to maintain a base image that is consistently updated, no?

[JulianHn]

I guess that is the sad reality. And thanks to Centos noch directly providing security info, the only way to get security updates in centos without finding out which updates are security patches is a full yum update, which will obviously also apply functionality updates. Not sure if and how this would affect OMERO.
So if we only want to apply security patches, it would probably be necessary to employ one of the docker image scanning tools that exist.

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/omero-production-server-current-best-practices/71074/4

[glyg]

Hi,
While moving on our projects, we realize that we'll need to build our own images for the omero services we'll deploy.

So I am re-upping @JulianHn question:

A more general question: Does it make sense to put much more work into the CentOS image? I assume that this Docker will probably switch over to a different distribution in the future (Alma ?), since CentOS changed its distribution model?

So if I am wondering whether I should base those image from opemicroscopy/omero-server or start a new Dockerfile from a standard alpine (or almalinux ?) base.
This could be the occasion to produce a more "optimized" docker image

I am more a apt than a yum person so would favor alpine but this is open to discussion

[joshmoore]

It's really a question for the larger community, but I at least could see reviewing a proposal for converting the main Dockerfile from centos to ubuntu.

[glyg]

ok, I'm giving it a shot and will report here