Errors when starting up containers for local development

[bfc5288]

Issue report

Users have reported errors when running the SCiMMA Server container during local development:

# source local-config.env; echo $ALERT_TOPIC; source run-containers.sh; hop subscribe -s EARLIEST --no-auth $ALERT_TOPIC
kafka://localhost:9092/snews-alert
766a41f9a039f6fdbc6e0ffd903d0b09b04d6befb3d4ae928c84cd1e9623801d
d24e63e3ebb0983f412688dd4fa7f05d4881af2cba34dd799bf74b215957fddb
%6|1612548695.721|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
    s.write(loader.load(message))
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
    return self._producer.__exit__(*exc)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
    n_unsent = self.close()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
    return self.flush()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
    n = self._producer.flush(timeout.total_seconds())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
    raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 5ms in state APIVERSION_QUERY)
%4|1612548695.749|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (108 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
%6|1612548696.185|FAIL|rdkafka#producer-1| [thrd:localhost:9092/bootstrap]: localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
internal kafka error: KafkaError{code=_TRANSPORT,val=-195,str="localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)"}
Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/publish.py", line 42, in _main
    s.write(loader.load(message))
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 343, in __exit__
    return self._producer.__exit__(*exc)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 59, in __exit__
    n_unsent = self.close()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 49, in close
    return self.flush()
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/producer.py", line 40, in flush
    n = self._producer.flush(timeout.total_seconds())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/errors.py", line 22, in log_client_errors
    raise(KafkaException.from_kafka_error(kafka_error))
adc.errors.KafkaException: Error communicating with Kafka: code=_TRANSPORT localhost:9092/bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api.version.request) (after 3ms in state APIVERSION_QUERY)
%4|1612548696.195|TERMINATE|rdkafka#producer-1| [thrd:app]: Producer terminating with 1 message (102 bytes) still in queue or transit: use flush() to wait for outstanding message delivery
Traceback (most recent call last):
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 36, in subscribe
    topic_meta = self.describe_topic(topic, timeout)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 61, in describe_topic
    return cluster_meta.topics[topic]
KeyError: 'snews-alert'During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/Users//snews/snews-venv/bin/hop", line 8, in <module>
    sys.exit(main())
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/__main__.py", line 54, in main
    args.func(args)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/subscribe.py", line 55, in _main
    with stream.open(args.url, "r") as s:
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 103, in open
    return _Consumer(
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/hop/io.py", line 246, in __init__
    self._consumer.subscribe(t)
  File "/Users//snews/snews-venv/lib/python3.8/site-packages/adc/consumer.py", line 38, in subscribe
    raise ValueError(f"topic {topic} does not exist on the broker, so can't subscribe")
ValueError: topic snews-alert does not exist on the broker, so can't subscribe

Cause

This is due to the SCiMMA Server container not fully loading before the start-up messages are published in run-containers.sh, causing ValueError: topic snews-alert does not exist on the broker, so can't subscribe issues when trying to access the topic.

Subsequent runs of run-containers.sh usually do not have this issue, since the container usually is fully loaded by then and can hence receive messages and create topics.

Possible solutions

A workaround would be to increase the sleep time in run-containers.sh so that the container loads before receiving the start-up messages.

A more robust solution would be to modify scimma-server-container to hold incoming message requests until the container has loaded.

An alternative could be to modify hop-client to create non-existing topics when subscribing, but this would be a substantial design change with potentially undesired effects.