Update GSC with Gramine's new SGX Driver Requirements

[adarshan-intel]

Description of the Changes

This pull request updates the Gramine Shielded Containers (GSC) to comply with the new SGX driver requirements.
As part of the gramineproject/gramine#2061 in the Gramine master branch, the options -Dsgx_driver, -Dsgx_driver_include_path, and -Dsgx_driver_device have been removed. Similar changes should be backported into GSC as well.

It includes modifications to the Dockerfile and entry point templates for CentOS and Debian. Additionally, it removes the -Dsgx_driver and -Dsgx_driver_include_path options when building Gramine inside the container. For further details, please refer to gramine PR #2061

Previously, we explicitly used to locate the sgx.h package in /gramine/driver using Dsgx_driver_include_path.
However, we are now installing additional package linux-libc-dev that provides sgx.h out of the box.

Fixes #227

How to Test This PR

Continuous Integration (CI)

---

This change is 

[adarshan-intel]

@kailun-qin Please review

[anirbanbasu]

These changes now result in meson.build:108:7: ERROR: C header 'asm/sgx.h' not found -- I am not sure what am I missing.

[adarshan-intel]

Can you go through this PR - Remove support for OOT (out-of-tree) SGX driver by woju · Pull Request #2061 · gramineproject/gramine
It has explanation of why this could be failing, probably older kernel?

Reviewable status: 0 of 4 files reviewed, all discussions resolved, not enough approvals from maintainers (2 more required), not enough approvals from different teams (2 more required, approved so far: ), "fixup! " found in commit messages' one-liners

[anirbanbasu]

Can you go through this PR - Remove support for OOT (out-of-tree) SGX driver by woju · Pull Request #2061 · gramineproject/gramine It has explanation of why this could be failing, probably older kernel?

Reviewable status: 0 of 4 files reviewed, all discussions resolved, not enough approvals from maintainers (2 more required), not enough approvals from different teams (2 more required, approved so far: ), "fixup! " found in commit messages' one-liners

Thanks @adarshan-intel, I am using the 6.8.0-40-generic kernel -- thus, asm/sgx.h should be available to meson on line 108 of the meson build (https://github.com/gramineproject/gramine/blob/5789620a6d04208e5803321953e09eca2c24dd7b/meson.build#L108), isn't it?

[adarshan-intel]

@anirbanbasu Have made a recent change in the commit, try running again.
Previously, we explicitly used to locate the sgx.h package in /gramine/driver using Dsgx_driver_include_path.
However, we are now including it in /usr/include, which contains all the header files by default.

[woju]

@woju Do you have an idea how to proceed with this?

The correct solution is to install linux-libc-dev package that needs to match kernel >= 5.11, provided that distro ships recent enough version (all distros supported by Gramine do). On which distros this error happens?

(Downloading the header from the Internet and copying it into /usr/include is not a correct solution).

[anirbanbasu]

@adarshan-intel, downloading to /usr/include does not throw the asm/sgx.h not found error. It builds but the built container does not execute -- it may be some other application-level error, which I need to investigate. I see that you have changed some other files but I have not tried those changes yet.

@woju, this is happening on Ubuntu and Debian. Should I install linux-libc-dev on my application container?

[anirbanbasu]

While the build process succeeds, I get a separate error when starting the built container as [P1:T1:] error: libos_init() failed in init_exec_handle: No such file or directory (ENOENT) but I am inclined to believe that this is some misconfiguration at my application level.

[woju]

Should I install linux-libc-dev on my application container?

Yes, you need to install it wherever you compile Gramine.

[anirbanbasu]

Should I install linux-libc-dev on my application container?

Yes, you need to install it wherever you compile Gramine.

@woju, should this (https://github.com/gramineproject/gsc/blob/master/templates/Dockerfile.common.compile.template) not be the one to install linux-libc-dev?

[anirbanbasu]

While the build process succeeds, I get a separate error when starting the built container as [P1:T1:] error: libos_init() failed in init_exec_handle: No such file or directory (ENOENT) but I am inclined to believe that this is some misconfiguration at my application level.

Just to make sure that this comment of mine does not confuse anyone, this turned out to be a misconfiguration in my application, which has now been dealt with.

[woju]

@woju, should this (https://github.com/gramineproject/gsc/blob/master/templates/Dockerfile.common.compile.template) not be the one to install linux-libc-dev?

It should be installed in templates that derive from this one, because package installation is different in different distributions (i.e. in this apt-get install:

gsc/templates/debian/Dockerfile.compile.template

Line 7 in e57a501

&& env DEBIAN_FRONTEND=noninteractive apt-get install -y \

and other distros have comparable commands that also need to be adjusted).

[adarshan-intel]

The latest commit e57fc97 (#228) does what @woju said, to remove instllation of sgx.h from internet and install from linux-libc-dev package that provides sgx.h,

I have tested basic runs on Ubuntu, Debian, CentOS, CentOS Stream 9, Red Hat, and SLES. Everything is running fine.

[anirbanbasu]

The latest commit e57fc97 (#228) does what @woju said, to remove instllation of sgx.h from internet and install from linux-libc-dev package that provides sgx.h,

I have tested basic runs on Ubuntu, Debian, CentOS, CentOS Stream 9, Red Hat, and SLES. Everything is running fine.

@adarshan-intel @woju in my case, this brought back the ERROR: C header 'asm/sgx.h' not found again, which can be fixed (again) by downloading sgx.h into /usr/include!

[adarshan-intel]

@anirbanbasu The solution I mentioned seems like a bit of a hack and might not be the best approach. According to @woju I've added the necessary package to the templates/debian/Dockerfile.compile.template Docker file, which includes the sgx.h header. Since you're using Ubuntu or debian, linux-libc-dev should already be installed in the latest commit when you run gsc build which contains sgx.h package

@woju Is there something I am missing here?

[adarshan-intel]

@anirbanbasu By the way, which Ubuntu version are you using? Is it 20.04, 22.04, 23.10 or 24.04?
If you are using Ubuntu 20.04 then this distro is not supported in further release cycle.

[anirbanbasu]

@anirbanbasu By the way, which Ubuntu version are you using? Is it 20.04, 22.04, 23.10 or 24.04? If you are using Ubuntu 20.04 then this distro is not supported in further release cycle.

@adarshan-intel I am using Ubuntu 22.04.5 LTS.

[adarshan-intel]

@anirbanbasu The changes I made should be working on Ubuntu 22.04.5. Everything looks good from my end, but let's wait for @woju's opinion to confirm.

[adarshan-intel]

@anirbanbasu Could you share the logs with me? Also, template files to troubleshoot the issue?

[anirbanbasu]

@anirbanbasu Could you share the logs with me? Also, template files to troubleshoot the issue?

@adarshan-intel unfortunately, due to confidentiality reasons, I cannot.

The template file templates/Dockerfile.common.compile.template looks exactly like the one you have on this pull request except for internal proxy configuration that should be enabled within my network to have git pull from the Gramine repository. I also modified each and every other file the way you have in your pull request.

Regarding the logs, if I remove the manual download of sgx.h into /usr/include/asm then I get the same error as the original one I reported, i.e., meson.build:108:7: ERROR: C header 'asm/sgx.h' not found.

[anirbanbasu]

@adarshan-intel Previously, this was downloading sgx.h from uapi/asm/sgx.h for x86. The linux-libc-dev package seems to have a number of these sgx.h (https://packages.debian.org/sid/all/linux-libc-dev/filelist). Could there be a conflict of includes? I am not too familiar with the internals of what's going on here, so I may be poking around in the dark.

[woju]

No, it's OK. Note they're in a directories named after compiler triplets (those things like x86_64-linux-gnu), so only one will be used at a time. It's fine.

Please use reviewable for discussions about PRs.

[woju]

Sorry to be blunt, but without seeing both logs and your actual changes (rather than relying on your descriptions of changes, because you might have made a typo), it's hard to tell anything about the reasons of the error you see. I've tested this PR's ./gsc build-gramine with debian:12, works fine with regard to <asm/sgx.h>, so unless proven otherwise, I'm going to assume the problem is on your side.

woju@[...] ~/src/gsc [git||remotes/origin/pull/228/head] [17:36 4295]% ./gsc build-gramine asdf
/usr/local/lib/python3.8/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
Building base-Gramine Docker image `asdf`...
Step 1/8 : FROM debian:12 AS gramine
[...]
Step 8/8 : RUN cd /gramine     && meson setup build/ --prefix="/gramine/meson_build_output"        --buildtype=release        -Ddirect=enabled -Dsgx=enabled             && ninja -C build     && ninja -C build install

 ---> Running in 46f06c28de1e
The Meson build system
[...]
Has header "asm/sgx.h" : YES
[...]
 ---> 4c336f858e3c
Successfully built 4c336f858e3c
Successfully tagged asdf:latest
Successfully built a base-Gramine Docker image `asdf`.

Or am I missing something?

Reviewed 2 of 3 files at r2, 9 of 9 files at r4, all commit messages.
Reviewable status: all files reviewed, 9 unresolved discussions, not enough approvals from maintainers (2 more required), not enough approvals from different teams (2 more required, approved so far: ), "fixup! " found in commit messages' one-liners (waiting on @adarshan-intel and @anirbanbasu)

---

templates/centos/Dockerfile.compile.template line 22 at r4 (raw file):

        git \
        httpd \
        kernel-devel \

This package needs to be kernel-headers. If you write kernel-devel, it will also install kernel-headers because of dependencies, (kernel-devel → gcc → glibc-devel → kernel-headers), but this is not what you want to write here.

Suggestion:

kernel-headers

---

templates/centos/Dockerfile.compile.template line 40 at r4 (raw file):

        python3-protobuf \
        rpm-build \
        wget \

I think wget is not used anywhere else, apart from the section that downloaded sgx.h, which you just removed, so you can also remove wget installation (here and everywhere else).

---

templates/suse/Dockerfile.compile.template line 29 at r4 (raw file):

        gcc11-c++ \
        git \
        kernel-devel \

On SUSE it's called linux-glibc-devel (https://software.opensuse.org/package/linux-glibc-devel)

Suggestion:

linux-glibc-devel

---

templates/debian/entrypoint.manifest.template line 4 at r4 (raw file):

{% block loader %}
loader.entrypoint = "file:/gramine/meson_build_output/lib/x86_64-linux-gnu/gramine/libsysdb.so"

ditto (put it in that second commit, together with the other entrypoint line)

---

templates/redhat/ubi-minimal/Dockerfile.compile.template line 28 at r4 (raw file):

        git \
        httpd \
        kernel-devel \

Suggestion:

kernel-headers

---

templates/centos/entrypoint.manifest.template line 4 at r4 (raw file):

{% block loader %}
loader.entrypoint = "file:/gramine/meson_build_output/lib64/gramine/libsysdb.so"

This is unrelated change, needs to be in a separate commit.

---

templates/centos/stream/Dockerfile.compile.template line 21 at r4 (raw file):

        git \
        httpd \
        kernel-devel \

Suggestion:

kernel-headers

---

templates/redhat/ubi/Dockerfile.compile.template line 28 at r4 (raw file):

        git \
        httpd \
        kernel-devel \

Suggestion:

kernel-headers

[woju]

No, it's OK. Note they're in a directories named after compiler triplets (those things like x86_64-linux-gnu), so only one will be used at a time. It's fine.

Please use reviewable for discussions about PRs.

[anirbanbasu]

Sorry to be blunt, but without seeing both logs and your actual changes (rather than relying on your descriptions of changes, because you might have made a typo), it's hard to tell anything about the reasons of the error you see.

@woju, yes, understandable. I will check things at my end.

Please use reviewable for discussions about PRs.

Okay!