Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Forced stop of tee-post-compute does not update the session in CAS and prevents retry #22

Open
zguesmi opened this issue Mar 16, 2021 · 0 comments
Open

Forced stop of tee-post-compute does not update the session in CAS and prevents retry #22

zguesmi opened this issue Mar 16, 2021 · 0 comments

Comments

@zguesmi
Copy link
Member

zguesmi commented Mar 16, 2021

When post-compute container is forced to stop (docker container rm -f) the next time we use the same task config
we get this error:

App stdout:

[SCONE|ERROR] src/process/init.c:332:_cas_get_fspf_secret(): Error receiving fspf key response from CAS: Error: Unable to receive key for fspf file from backend
Caused by: You have insufficient rights to access the fspf key (the supplied tag is not fresh - suspected replay attack)

Post compute stdout:

[INFO ] 2020-07-27 14:01:15.991 [main] App - Tee worker post-compute started
[INFO ] 2020-07-27 14:01:15.999 [main] FlowManager - ReadComputedFile stage started
[ERROR] 2020-07-27 14:01:16.010 [main] FileHelper - Failed to read file [filePath:/iexec_out/computed.json]
[ERROR] 2020-07-27 14:01:16.011 [main] IexecFileHelper - Failed to read compute file (invalid path)

Complete logs:

App

export SCONE_QUEUES=4
export SCONE_SLOTS=256
export SCONE_SIGPIPE=0
export SCONE_MMAP32BIT=0
export SCONE_SSPINS=100
export SCONE_SSLEEP=4000
export SCONE_LOG=7
export SCONE_HEAP=1073741824
export SCONE_STACK=2097152
export SCONE_CONFIG=/etc/sgx-musl.conf
export SCONE_ESPINS=10000
export SCONE_MODE=hw
export SCONE_ALLOW_DLOPEN=yes (unprotected)
export SCONE_MPROTECT=yes
musl version: 1.1.24
Revision: f4655f36e58c3f91fd4e085285c3a163a841e802 (Thu Apr 30 11:19:42 2020 +0000)
Branch: HEAD
Enclave hash: 2a421b3b7a6f771c3a602f49ce05b6a75793312b8e2c61c673fe7085a16cf138
[SCONE|INFO] src/shielding/crypto.c:222:crypto_get_identity(): Generated enclave certificate
[SCONE|INFO] src/shielding/eai_attestor.c:174:eai_attestor_init(): Created TLS context to communicate with CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:288:eai_attestor_attest(): Sending Attestation Request to LAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:311:eai_attestor_attest(): Got Quote from LAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:324:eai_attestor_attest(): Sending enclave hello message to CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:355:eai_attestor_attest(): Got Attestation Request from CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:373:eai_attestor_attest(): Sending Attestation Request to LAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:396:eai_attestor_attest(): Got Quote from LAS: Forwarding it to CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:349:eai_attestor_attest(): Successfully attested enclave via CAS
[SCONE|DEBUG] src/process/init.c:639:__scone_prepare_secure_config(): Sending configuration request to CAS!
[SCONE|DEBUG] src/process/init.c:666:__scone_prepare_secure_config(): Received configuration from CAS!
[SCONE|DEBUG] src/process/init.c:323:_cas_get_fspf_secret(): Request volume secrets for /fspf.pb
[SCONE|DEBUG] src/process/init.c:324:_cas_get_fspf_secret(): Local volume state:
    [57 5f f4 5f 37 13 9f 6c 9a dd 0c 72 e9 61 13 f2 ]
[SCONE|ERROR] src/process/init.c:332:_cas_get_fspf_secret(): Error receiving fspf key response from CAS: Error: Unable to receive key for fspf file from backend
Caused by: You have insufficient rights to access the fspf key (the supplied tag is not fresh - suspected replay attack)
...

Tee post-compute

export SCONE_QUEUES=4
export SCONE_SLOTS=256
export SCONE_SIGPIPE=0
export SCONE_MMAP32BIT=0
export SCONE_SSPINS=100
export SCONE_SSLEEP=4000
export SCONE_LOG=7
export SCONE_HEAP=3221225472
export SCONE_STACK=2097152
export SCONE_CONFIG=/etc/sgx-musl.conf
export SCONE_ESPINS=10000
export SCONE_MODE=hw
export SCONE_ALLOW_DLOPEN=yes (unprotected)
export SCONE_MPROTECT=no
musl version: 1.1.24
Revision: f4655f36e58c3f91fd4e085285c3a163a841e802 (Thu Apr 30 11:19:42 2020 +0000)
Branch: HEAD

Enclave hash: 13076027fc67accba753a3ed2edf03227dfd013b450d68833a5589ec44132100
[SCONE|INFO] src/shielding/crypto.c:222:crypto_get_identity(): Generated enclave certificate
[SCONE|INFO] src/shielding/eai_attestor.c:174:eai_attestor_init(): Created TLS context to communicate with CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:288:eai_attestor_attest(): Sending Attestation Request to LAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:311:eai_attestor_attest(): Got Quote from LAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:324:eai_attestor_attest(): Sending enclave hello message to CAS
[SCONE|DEBUG] src/shielding/eai_attestor.c:349:eai_attestor_attest(): Successfully attested enclave via CAS
[SCONE|DEBUG] src/process/init.c:639:__scone_prepare_secure_config(): Sending configuration request to CAS!
[SCONE|DEBUG] src/process/init.c:666:__scone_prepare_secure_config(): Received configuration from CAS!
[SCONE|DEBUG] src/process/init.c:323:_cas_get_fspf_secret(): Request volume secrets for /fspf.pb
[SCONE|DEBUG] src/process/init.c:324:_cas_get_fspf_secret(): Local volume state:
    [1e b6 27 c1 c9 4b bc a0 31 78 b0 99 b1 3f b4 d1 ]
[SCONE|DEBUG] src/process/init.c:341:_cas_get_fspf_secret(): Received volume secrets successfully
[SCONE|DEBUG] src/process/init.c:676:__scone_prepare_secure_config(): Requesting Volume Secrets from CAS
[SCONE|DEBUG] src/process/init.c:323:_cas_get_fspf_secret(): Request volume secrets for /iexec_out/volume.fspf
[SCONE|DEBUG] src/process/init.c:324:_cas_get_fspf_secret(): Local volume state:
    [00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]
[SCONE|DEBUG] src/process/init.c:341:_cas_get_fspf_secret(): Received volume secrets successfully
[SCONE|DEBUG] src/process/init.c:323:_cas_get_fspf_secret(): Request volume secrets for /scone/volume.fspf
[SCONE|DEBUG] src/process/init.c:324:_cas_get_fspf_secret(): Local volume state:
    [00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ]
[SCONE|DEBUG] src/process/init.c:341:_cas_get_fspf_secret(): Received volume secrets successfully
[SCONE|DEBUG] src/process/init.c:687:__scone_prepare_secure_config(): Received Volume Secrets from CAS!
[SCONE|DEBUG] src/shielding/hierarchy.c:96:fs_hierarchy_update_tag(): Updated fspf tag of /iexec_out/volume.fspf to 37B43B505EC4D2DFAC96B540A7F99E53
[SCONE|DEBUG] src/process/init.c:794:__scone_apply_secure_config(): Loading FSPF from /iexec_out/volume.fspf with expected tag 37B43B505EC4D2DFAC96B540A7F99E53
[SCONE|DEBUG] src/shielding/hierarchy.c:96:fs_hierarchy_update_tag(): Updated fspf tag of /scone/volume.fspf to 0C714BA814A2D8749C9A01A0C127D36A
[SCONE|DEBUG] src/process/init.c:794:__scone_apply_secure_config(): Loading FSPF from /scone/volume.fspf with expected tag 0C714BA814A2D8749C9A01A0C127D36A
[SCONE|WARN] src/syscall/syscall.c:698:__scone_syscall_unshielded(): system call: SYS_membarrier, number 324 is not implemented.
[SCONE|WARN] src/syscall/syscall.c:698:__scone_syscall_unshielded(): system call: SYS_membarrier, number 324 is not implemented.
Picked up JAVA_TOOL_OPTIONS: -Xmx256m
[SCONE|WARN] src/shielding/proc_fs.c:368:_proc_fs_open(): open: /proc/self/mountinfo is not supported
[SCONE|WARN] src/shielding/proc_fs.c:368:_proc_fs_open(): open: /proc/self/coredump_filter is not supported
[SCONE|WARN] src/shielding/proc_fs.c:368:_proc_fs_open(): open: /proc/self/coredump_filter is not supported
[SCONE|WARN] src/syscall/syscall.c:698:__scone_syscall_unshielded(): system call: SYS_membarrier, number 324 is not implemented.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
[SCONE|WARN] src/syscall/anon.c:144:mmap_anon(): Protected heap memory exhausted! Set SCONE_HEAP environment variable to increase it.
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
[SCONE|WARN] src/shielding/proc_fs.c:368:_proc_fs_open(): open: /proc/net/ipv6_route is not supported
[INFO ] 2020-07-27 14:01:15.991 [main] App - Tee worker post-compute started
[INFO ] 2020-07-27 14:01:15.999 [main] FlowManager - ReadComputedFile stage started
[ERROR] 2020-07-27 14:01:16.010 [main] FileHelper - Failed to read file [filePath:/iexec_out/computed.json]
[ERROR] 2020-07-27 14:01:16.011 [main] IexecFileHelper - Failed to read compute file (invalid path)[chainTaskId:0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246, computedFilePath:/iexec_out/]
[ERROR] 2020-07-27 14:01:16.012 [main] FlowManager - ReadComputedFile failed (computed.json missing)

2020-07-27 14:01:16.927  INFO 6 --- [         task-4] com.iexec.worker.compute.ComputeService  : Saved stdout file [path:/tmp/iexec-worker/worker-sgx/0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246/output/stdout.txt]
2020-07-27 14:01:16.942 ERROR 6 --- [         task-4] com.iexec.common.utils.FileHelper        : Failed to read file [filePath:/tmp/iexec-worker/worker-sgx/0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246/output/computed.json]
2020-07-27 14:01:16.944 ERROR 6 --- [         task-4] com.iexec.common.utils.IexecFileHelper   : Failed to read compute file (invalid path)[chainTaskId:0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246, computedFilePath:/tmp/iexec-worker/worker-sgx/0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246/output/]
2020-07-27 14:01:16.944 ERROR 6 --- [         task-4] com.iexec.worker.compute.ComputeService  : Failed to getComputedFile (computed.json missing)[chainTaskId:0x05c1267e3774f8d6b42c325a60270fe07da5766d6c698699b52fbaa82baef246]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zguesmi