Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Appearance of messy code during connection between server in Occlum and client in host #99

Open
fengyuanyu1 opened this issue Aug 4, 2022 · 0 comments
Assignees

Comments

@fengyuanyu1
Copy link

I try to execute the sample server in Occlum and client out of enclave. I find that they can connect successfully but the messages delivered by server in Occlum has some unreadable code.

Reproduce

Double containers:

You can reproduce using the following Dockerfiles:
The first one is Dockerfile for rats-server in Occlum:

#rats_server-Dockerfile
FROM docker.io/occlum/occlum:0.27.3-ubuntu20.04
RUN sed -i '2,2s/localhost:8081/sgx-dcap-server.cn-beijing.aliyuncs.com/g' /etc/sgx_default_qcnl.conf &&\
    sed -i '5,5s/#//' /etc/sgx_default_qcnl.conf &&\
    sed -i '5,5s/FALSE/TRUE/g' /etc/sgx_default_qcnl.conf
RUN echo "Downloading rats-tls" && \
    git clone https://github.com/inclavare-containers/rats-tls.git
WORKDIR /root/rats-tls
RUN cmake -DRATS_TLS_BUILD_MODE="occlum" -DBUILD_SAMPLES=on -H. -Bbuild &&\
    make -C build install

The second one is Dockerfile for rats-client in Occlum:

#rats_client-Dockerfile
FROM docker.io/occlum/occlum:0.27.3-ubuntu20.04
RUN sed -i '2,2s/localhost:8081/sgx-dcap-server.cn-beijing.aliyuncs.com/g' /etc/sgx_default_qcnl.conf &&\
    sed -i '5,5s/#//' /etc/sgx_default_qcnl.conf &&\
    sed -i '5,5s/FALSE/TRUE/g' /etc/sgx_default_qcnl.conf
RUN echo "Downloading rats-tls" && \
    git clone https://github.com/inclavare-containers/rats-tls.git
WORKDIR /root/rats-tls
RUN cmake -DBUILD_SAMPLES=on -H. -Bbuild &&\
    make -C build install

The run the commands below to obtain two running containers.

docker build -f rats_server-Dockerfile -t rats_test:sv0.1 .
docker build -f rats_client-Dockerfile -t rats_test:cv0.1 .
docker run -itd --name rte_server --privileged -v /dev/sgx_enclave:/dev/sgx/enclave -v /dev/sgx_provision:/dev/sgx/provision --net=host rats_test:sv0.1
docker run -itd --name rte_client --privileged --net=host rats_test:cv0.1

Enter the corresponding container and launch the server and client.

#rte_server
cd /usr/share/rats-tls/samples
rm -rf occlum_workspace_server
mkdir occlum_workspace_server
cd occlum_workspace_server
occlum init
cp ../rats-tls-server image/bin
cp /lib/x86_64-linux-gnu/libdl.so.2 image/opt/occlum/glibc/lib
cp /usr/lib/x86_64-linux-gnu/libssl.so.1.1 image/opt/occlum/glibc/lib
cp /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 image/opt/occlum/glibc/lib
mkdir -p image/usr/local/lib
cp -rf /usr/local/lib/rats-tls image/usr/local/lib
occlum build
occlum run /bin/rats-tls-server -m -l debug -a sgx_ecdsa -v nullverifier

#rte_client
cd /usr/share/rats-tls/samples
./rats-tls-client -l debug -m -a nullattester -v sgx_ecdsa

You can obtain the similar output like the next two code boxes:

#rte_server
root@iZ2ze9ypcfpk5tv77g5tx6Z:/usr/share/rats-tls/samples/occlum_workspace_server# occlum run /bin/rats-tls-server -m -l debug -a sgx_ecdsa -v nullverifier 
    - Welcome to RATS-TLS sample server program for Occlum
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rats_tls_init()@L22: conf 0x7f19a94217e0, handle 0x7f19a9421770
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_crypto_wrapper_select()@L27: selecting the crypto wrapper '(null)' ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_init()@L16: ctx 0x7f19a702ebc0
[INFO] the crypto wrapper 'openssl' selected
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_attester_select()@L33: selecting the enclave attester 'sgx_ecdsa' cert algo '0x1'...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] init_enclave_attester()@L17: called enclave core ctx: 0xa7024570 enclave attester ctx: 0xa7024680 algo: 0x1
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] sgx_ecdsa_attester_init()@L14: ctx 0x7f19a7024680, algo 1
[INFO] the enclave attester 'sgx_ecdsa' selected
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_verifier_select()@L32: selecting the enclave verifier 'nullverifier' ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_verifier_select()@L36: trying to match sgx_ecdsa_qve ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_verifier_select()@L36: trying to match nullverifier ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] init_enclave_verifier()@L17: init enclave verifier rtls_core_context: 0xa7024570
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] nullverifier_init()@L14: ctx 0x7f19a7024710, algo 1
[INFO] the enclave verifier 'nullverifier' selected
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_tls_wrapper_select()@L28: selecting the tls wrapper '(null)' ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_tls_init()@L14: ctx 0x7f19a7024780
[INFO] the tls wrapper 'openssl' selected
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] rtls_core_generate_certificate()@L15: ctx 0x7f19a7024570
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_gen_privkey()@L20: ctx 0x7f19a702ebc0, algo 1, privkey_buf 0x7f19a9420de0, privkey_len 0x7f19a940cd20
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_gen_privkey()@L29: 2048-byte private key buffer requested ...
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_gen_privkey()@L71: ECC-256 private key (121-byte) in DER format generated
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_gen_pubkey_hash()@L18: ctx 0x7f19a702ebc0, algo 1, hash 0x7f19a940ccc0
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] openssl_gen_pubkey_hash()@L34: the sha256 of public key [91] 911a206b5e21bc59...10426abb
Thu Aug  4 12:41:22 UTC 2022: [DEBUG] sgx_ecdsa_collect_evidence()@L70: ctx 0x7f19a7024680, evidence 0x7f19a940ed7c, algo 1, hash 0x7f19a940ccc0
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] sgx_ecdsa_collect_evidence()@L125: Succeed to generate the quote!
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] openssl_gen_cert()@L113: ctx 0x7f19a702ebc0, cert_info 0x7f19a940cd60
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] openssl_gen_cert()@L166: evidence type 'sgx_ecdsa' requested
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] openssl_gen_cert()@L229: self-signing certificate generated
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] openssl_tls_use_privkey()@L14: ctx 0x7f19a7024780, privkey_buf 0x7f19a9420de0, privkey_len 121
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] openssl_tls_use_cert()@L13: ctx 0x7f19a7024780, cert_info 0x7f19a940cd60
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] rats_tls_init()@L106: the handle 0x7f19a7024570 returned
Thu Aug  4 12:41:23 UTC 2022: [DEBUG] rats_tls_set_verification_callback()@L12: set user verification callback handle: 0x7f19a9421770, cb (nil)
[INFO] Waiting for a connection ...
Thu Aug  4 12:41:51 UTC 2022: [DEBUG] rats_tls_negotiate()@L15: handle 0x7f19a7024570, fd 5
Thu Aug  4 12:41:51 UTC 2022: [DEBUG] openssl_tls_negotiate()@L110: ctx 0x7f19a7024780, fd 5
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] calc_pubkey_hash()@L101: The hash of public key [91] 9bd13565b3a3f223...2303ba33
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
[WARN] Unhandled evidence type nullverifier
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] tls_wrapper_verify_certificate_extension()@L19: tls_wrapper_verify_certificate_extension() called with evidence type: 'nullverifier'
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] nullverifier_verify_evidence()@L14: ctx 0x7f19a7024710, evidence 0x7f19a940f140, hash 0x7f19a94211a0
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] calc_pubkey_hash()@L101: The hash of public key [91] 9bd13565b3a3f223...2303ba33
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] find_oid()@L120: there are no extensions in X509 cert
[WARN] Unhandled evidence type nullverifier
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] tls_wrapper_verify_certificate_extension()@L19: tls_wrapper_verify_certificate_extension() called with evidence type: 'nullverifier'
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] nullverifier_verify_evidence()@L14: ctx 0x7f19a7024710, evidence 0x7f19a940f110, hash 0x7f19a9421170
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] openssl_internal_negotiate()@L92: success to negotiate
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] rats_tls_server_startup()@L211: Client connected successfully
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] rats_tls_receive()@L16: handle 0x7f19a7024570, buf 0x7f19a9421a50, buf_size 0x7f19a9421780 (256-byte)
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] openssl_tls_receive()@L13: ctx 0x7f19a7024780, buf 0x7f19a9421a50, buf_size 0x7f19a9421780
[INFO] Client: Hello and welcome to RATS-TLS!

Thu Aug  4 12:41:53 UTC 2022: [DEBUG] rats_tls_transmit()@L16: handle 0x7f19a7024570, buf 0x7f19a9421a50, buf_size 0x7f19a9421780 (95-byte)
Thu Aug  4 12:41:53 UTC 2022: [DEBUG] openssl_tls_transmit()@L13: ctx 0x7f19a7024780, buf 0x7f19a9421a50, buf_size 0x7f19a9421780
[INFO] Waiting for a connection ...
```shell

```shell
#rte_client
root@iZ2ze9ypcfpk5tv77g5tx6Z:/usr/share/rats-tls/samples# ./rats-tls-client -l debug -m -a nullattester -v sgx_ecdsa
[ERROR] failed to read msr 0xc0010131
[ERROR] failed to read msr 0xc0010131
    - Welcome to RATS-TLS sample client program
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rats_tls_init()@L22: conf 0x7ffce2159ed0, handle 0x7ffce2159e70
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_crypto_wrapper_select()@L27: selecting the crypto wrapper '(null)' ...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_init()@L16: ctx 0x558ea7605980
[INFO] the crypto wrapper 'openssl' selected
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_attester_select()@L33: selecting the enclave attester 'nullattester' cert algo '0x1'...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] init_enclave_attester()@L17: called enclave core ctx: 0xa7605890 enclave attester ctx: 0xa76059c0 algo: 0x1
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] nullattester_init()@L14: ctx 0x558ea76059c0, algo 1
[INFO] the enclave attester 'nullattester' selected
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_verifier_select()@L32: selecting the enclave verifier 'sgx_ecdsa' ...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_verifier_select()@L36: trying to match sgx_ecdsa ...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] init_enclave_verifier()@L17: init enclave verifier rtls_core_context: 0xa7605890
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] sgx_ecdsa_verifier_init()@L14: ctx 0x558ea7605a30, algo 1
[INFO] the enclave verifier 'sgx_ecdsa' selected
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_tls_wrapper_select()@L28: selecting the tls wrapper '(null)' ...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_tls_init()@L14: ctx 0x558ea7605aa0
[INFO] the tls wrapper 'openssl' selected
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rtls_core_generate_certificate()@L15: ctx 0x558ea7605890
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_privkey()@L20: ctx 0x558ea7605980, algo 1, privkey_buf 0x7ffce21594f0, privkey_len 0x7ffce2145430
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_privkey()@L29: 2048-byte private key buffer requested ...
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_privkey()@L71: ECC-256 private key (121-byte) in DER format generated
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_pubkey_hash()@L18: ctx 0x558ea7605980, algo 1, hash 0x7ffce21453d0
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_pubkey_hash()@L34: the sha256 of public key [91] baa8e9e3d5947e90...5f6b39a8
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] nullattester_collect_evidence()@L15: ctx 0x558ea76059c0, evidence 0x7ffce214748c, algo 1, hash 0x7ffce21453d0
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_cert()@L113: ctx 0x558ea7605980, cert_info 0x7ffce2145470
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_cert()@L166: evidence type '' requested
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_gen_cert()@L229: self-signing certificate generated
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_tls_use_privkey()@L14: ctx 0x558ea7605aa0, privkey_buf 0x7ffce21594f0, privkey_len 121
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_tls_use_cert()@L13: ctx 0x558ea7605aa0, cert_info 0x7ffce2145470
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rats_tls_init()@L106: the handle 0x558ea7605890 returned
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rats_tls_set_verification_callback()@L12: set user verification callback handle: 0x7ffce2159e70, cb (nil)
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] rats_tls_negotiate()@L15: handle 0x558ea7605890, fd 3
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] openssl_tls_negotiate()@L110: ctx 0x558ea7605aa0, fd 3
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] calc_pubkey_hash()@L101: The hash of public key [91] 4d22caf6505fabfd...758b9fe3
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] tls_wrapper_verify_certificate_extension()@L19: tls_wrapper_verify_certificate_extension() called with evidence type: 'sgx_ecdsa'
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] sgx_ecdsa_verify_evidence()@L135: ctx 0x558ea7605a30, evidence 0x7ffce2147800, hash 0x7ffce2159860
Thu Aug  4 12:56:39 UTC 2022: [DEBUG] ecdsa_verify_evidence()@L57: quote size is 4729, quote signature_data_len is 4293
[get_driver_type edmm_utility.cpp:111] Failed to open Intel SGX device.
[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-trunk-215.1/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.
[INFO] sgx qv gets quote supplemental data size successfully.
[get_driver_type edmm_utility.cpp:111] Failed to open Intel SGX device.
[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-trunk-215.1/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.
[INFO] sgx qv verifies quote successfully.
[INFO] verification completed successfully.
Thu Aug  4 12:56:40 UTC 2022: [DEBUG] calc_pubkey_hash()@L101: The hash of public key [91] 4d22caf6505fabfd...758b9fe3
Thu Aug  4 12:56:40 UTC 2022: [DEBUG] tls_wrapper_verify_certificate_extension()@L19: tls_wrapper_verify_certificate_extension() called with evidence type: 'sgx_ecdsa'
Thu Aug  4 12:56:40 UTC 2022: [DEBUG] sgx_ecdsa_verify_evidence()@L135: ctx 0x558ea7605a30, evidence 0x7ffce21477d0, hash 0x7ffce2159830
Thu Aug  4 12:56:40 UTC 2022: [DEBUG] ecdsa_verify_evidence()@L57: quote size is 4729, quote signature_data_len is 4293
[get_driver_type edmm_utility.cpp:111] Failed to open Intel SGX device.
[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-trunk-215.1/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.
[INFO] sgx qv gets quote supplemental data size successfully.
[get_driver_type edmm_utility.cpp:111] Failed to open Intel SGX device.
[get_driver_type /home/sgx/jenkins/ubuntuServer2004-release-build-trunk-215.1/build_target/PROD/label/Builder-UbuntuSrv20/label_exp/ubuntu64/linux-trunk-opensource/psw/urts/linux/edmm_utility.cpp:111] Failed to open Intel SGX device.
[INFO] sgx qv verifies quote successfully.
[INFO] verification completed successfully.
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] openssl_internal_negotiate()@L94: success to connect
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] rats_tls_transmit()@L16: handle 0x558ea7605890, buf 0x558ea69ea1a8, buf_size 0x7ffce2159e80 (31-byte)
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] openssl_tls_transmit()@L13: ctx 0x558ea7605aa0, buf 0x558ea69ea1a8, buf_size 0x7ffce2159e80
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] rats_tls_receive()@L16: handle 0x558ea7605890, buf 0x7ffce2159f90, buf_size 0x7ffce2159e80 (256-byte)
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] openssl_tls_receive()@L13: ctx 0x558ea7605aa0, buf 0x7ffce2159f90, buf_size 0x7ffce2159e80
Server: 6��E��
#�^���}��p���*MwCș�i�p��ΞHello and welcome to RATS-TLS!

[ERROR] Invalid response retrieved from rats-tls server
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] rats_tls_cleanup()@L20: handle 0x558ea7605890
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] openssl_tls_cleanup()@L13: ctx 0x558ea7605aa0
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] nullattester_cleanup()@L12: called
Thu Aug  4 12:56:41 UTC 2022: [DEBUG] sgx_ecdsa_verifier_cleanup()@L13: called

Single container:

You can reproduce using the following commands: (refs: #93 )

# Build rats-tls in sgx mode to get rats-tls-server running in enclave
cmake -DRATS_TLS_BUILD_MODE="occlum" -DBUILD_SAMPLES=on -H. -Bbuild
make & make install

# Backup sgx mode programs
cp -a /usr/share/rats-tls/samples /usr/share/rats-tls/samples-sgx

# Build rats-tls in host mode to get rats-tls-client running on host
cmake -DRATS_TLS_BUILD_MODE="host" -DBUILD_SAMPLES=on -H. -Bbuild
make & make install

# Have a test
cd /usr/share/rats-tls/samples-sgx
./rats-tls-server --attester sgx_ecdsa --verifier nullverifier -l debug -m &
cd /usr/share/rats-tls/samples
./rats-tls-client --attester nullattester --verifier sgx_ecdsa -l debug -m

@haosanzi

@haosanzi haosanzi self-assigned this Aug 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants