Kurtosis zkevm-stateless-executor-001 stopped

[ArrjunPradeep]

System information

Ubuntu22.04

Commit id

d

Description & steps to reproduce

I'm getting the following error while executing this command "kurtosis run --enclave cdk ." [Deploy to Sepolia]

Desired behavior

Network running

What is the severity of this bug?

Critical; I am blocked and Kurtosis CDK is unusable for me because of this bug.

[praetoriansentry]

I don't think I've seen this before. It looks like the machine this is running on is very slow. As a result, the executor is taking so long to start that Kurtosis is assuming the process isn't starting properly.

What type of machine is this running on? I'll paste my logs below so you can compare, but it looks like the process in your screenshot is 10x slower

[ArrjunPradeep]

Windows WSL2 -> Ubuntu22.04

[praetoriansentry]

One option would be to add a wait attribute to the PortSpecs here:

kurtosis-cdk/lib/zkevm_prover.star

Lines 58 to 59 in 23694ed

	"hash-db": PortSpec(args["zkevm_hash_db_port"], application_protocol="grpc"),
	"executor": PortSpec(args["zkevm_executor_port"], application_protocol="grpc"),

Setting wait = None should fix the issue, but could cause issues with other components that expect the executor to be running.

Setting wait = 10m or some longer amount of time should give your system more time to load. The risk is that it's very slow or there is something else wrong that will make it take a very long time. I would experiment with both and see if you can get anywhere.

[ArrjunPradeep]

No, it doesn't work as expected. Encountered with the same error mentioned in the earlier thread above. According to official docs, If you notice some services, such as the zkevm-stateless-executor or zkevm-prover, consistently having the status of STOPPED, try increasing the Docker memory allocation. May I know how much it should be ?

[praetoriansentry]

That's strange. If you changed the wait time, you shouldn't get the same error. It would make sense if it never finished.. But strange if you're getting the same exact error as above.

I would guess you probably want at least 4GB. I'm running 3 networks right now and it looks like the biggest container is the zkProver process which is using roughly 2.5 GB

It's possible the max memory goes higher than that when it's initializing.

[leovct]

I recommend setting it to at least 10/12 GB, it may use less when running idle but when initialising, it requires a lot of memory. The different zkevm provers (zkevm-prover, zkevm-stateless-executor) use around 2.5 GB and the agglayer prover around 1GB. If you add all the other components, it quickly adds up.

[leovct]

Hello @ArrjunPradeep, have you managed to fix the issue?

[ArrjunPradeep]

Hi @leovct , I have set to 12GB around, Now its showing a different error :

[leovct]

Hi @leovct , I have set to 12GB around, Now its showing a different error :

It looks like you Docker daemon stopped. You might have to restart it. Then you'll need to clean the enclaves with kurtosis clean --all before deploying the enclave with kurtosis run --enclave cdk .

[ArrjunPradeep]

I tried cleaning the enclaves and starting again. But still :

[leovct]

Can you pull the latest changes?

[ArrjunPradeep]

Wait time is 20m :
ports = {
"hash-db": PortSpec(args["zkevm_hash_db_port"], application_protocol="grpc", wait='20m'),
"executor": PortSpec(args["zkevm_executor_port"], application_protocol="grpc", wait='20m'),
}

[leovct]

As @praetoriansentry mentioned, it looks like your machine is very slow.... Even after 20 minutes, the prover is still not started. I would advise to allocate more memory to docker, as suggested above.

[ArrjunPradeep]

@leovct Now I am running on macOS. Still I am getting error :

[leovct]

I would suggest to clean the enclave and give it a retry.