Simulator attempts to get genesis block prematurely before Besu finished starting

[ensi321]

This issue is discovered as part of the effort of fixing up the failed consensus tests on Besu hyperledger/besu#5689

Description

Occasionally, simulator fails to get genesis block in a test by showing the following error message:
can't get genesis: Post "http://172.17.0.9:8545": dial tcp 172.17.0.9:8545: connect: connection refused
This is due to api call was made while Besu was still initializing. This appears to happen at random, a failed test that has this error in current run may pass in the next run. I am unable to reproduce this issue on my local machine. The associated failed tests on https://hivetests.ethdevops.io ran fine on my local.

Example

pointMulAdd2_d31g0v0_London on Jul 6 run (log) failed but passed on Jul 12 run (log)
envInfo_d3g0v0_Merge passed on Jul 6 (log) but failed on Jul 12 (log)

Normally after Besu starts, there should be a message Ethereum main loop is up. indicating it is ready to receive rpc requests. This is shown on Jul 12 log, but Jul 6 log indicates Besu was abruptly stopped while it was still initializing.

Speculation

It could be StartClient() did not correctly determine if Besu is ready to receive requests. The retry mechanism in getBlock() did get triggered but failed on both first and second attempt and there lacked a waiting time between the two attempts

[fjl]

Hive polls the client container's TCP port 8545 every 100ms until it comes up, then signals to the test the client is ready.