Bug in workers: "Signature expired"

[msrb]

In prod:

{"task_name": "dependency_snapshot", "event": "TASK_END", "task_id": "38678a6a-cf23-49c3-8acb-3b6543ebd0d4", "node_args": {"_release": "maven:com.manywho.sdk:sdk-server-servlet3:2.0.0-beta-04", "ecosystem": "maven", "_audit": {"version": "v1", "started_at": "2018-02-22T11:25:50.268175", "ended_at": "2018-02-22T11:25:51.339795"}, "job_id": "mavenReleasesJob", "name": "com.manywho.sdk:sdk-server-servlet3", "version": "2.0.0-beta-04", "document_id": 5121261, "recursive_limit": 0}, "storage": "BayesianPostgres", "queue": "prod_ingestion_DependencySnapshotTask_v0", "flow_name": "bayesianAnalysisFlow", "parent": {"metadata": "7e881a2e-3393-42b9-8635-34202f0af974"}, "dispatcher_id": "00074ee5-058c-4e34-8b87-c525ae91b106"}
2018-02-22 20:59:58,654 [INFO] celery.app.trace: Task selinon.SelinonTaskEnvelope[38678a6a-cf23-49c3-8acb-3b6543ebd0d4] succeeded in 0.09870916046202183s: None
2018-02-22 21:00:01,004 [CRITICAL] celery.worker: Unrecoverable error: Exception('Request HTTP Error  HTTP 403  Forbidden (b\'<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Signature expired: 20180222T204115Z is now earlier than 20180222T204501Z (20180222T210001Z - 15 min.)</Message><Detail/></Error><RequestId>cc2bfe62-b17a-5809-bb62-ad6b80eb846a</RequestId></ErrorResponse>\')',)
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/lib/python3.4/site-packages/celery/bootsteps.py", line 370, in start
    return self.obj.start()
  File "/usr/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 320, in start
    blueprint.start(self)
  File "/usr/lib/python3.4/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/usr/lib/python3.4/site-packages/celery/worker/consumer/consumer.py", line 596, in start
    c.loop(*c.loop_args())
  File "/usr/lib/python3.4/site-packages/celery/worker/loops.py", line 88, in asynloop
    next(loop)
  File "/usr/lib/python3.4/site-packages/kombu/async/hub.py", line 354, in create_loop
    cb(*cbargs)
  File "/usr/lib/python3.4/site-packages/kombu/async/http/curl.py", line 111, in on_readable
    return self._on_event(fd, _pycurl.CSELECT_IN)
  File "/usr/lib/python3.4/site-packages/kombu/async/http/curl.py", line 124, in _on_event
    self._process_pending_requests()
  File "/usr/lib/python3.4/site-packages/kombu/async/http/curl.py", line 130, in _process_pending_requests
    self._process(curl)
  File "/usr/lib/python3.4/site-packages/kombu/async/http/curl.py", line 178, in _process
    buffer=buffer, effective_url=effective_url, error=error,
  File "/usr/lib/python3.4/site-packages/vine/promises.py", line 150, in __call__
    svpending(*ca, **ck)
  File "/usr/lib/python3.4/site-packages/vine/promises.py", line 143, in __call__
    return self.throw()
  File "/usr/lib/python3.4/site-packages/vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/lib/python3.4/site-packages/vine/funtools.py", line 100, in _transback
    return callback(ret)
  File "/usr/lib/python3.4/site-packages/vine/promises.py", line 143, in __call__
    return self.throw()
  File "/usr/lib/python3.4/site-packages/vine/promises.py", line 140, in __call__
    retval = fun(*final_args, **final_kwargs)
  File "/usr/lib/python3.4/site-packages/vine/funtools.py", line 98, in _transback
    callback.throw()
  File "/usr/lib/python3.4/site-packages/vine/funtools.py", line 96, in _transback
    ret = filter_(*args + (ret,), **kwargs)
  File "/usr/lib/python3.4/site-packages/kombu/async/aws/connection.py", line 233, in _on_list_ready
    raise self._for_status(response, response.read())
Exception: Request HTTP Error HTTP 403 Forbidden (b'<?xml version="1.0"?><ErrorResponse xmlns="http://queue.amazonaws.com/doc/2012-11-05/"><Error><Type>Sender</Type><Code>SignatureDoesNotMatch</Code><Message>Signature expired: 20180222T204115Z is now earlier than 20180222T204501Z (20180222T210001Z - 15 min.)</Message><Detail/></Error><RequestId>cc2bfe62-b17a-5809-bb62-ad6b80eb846a</RequestId></ErrorResponse>')

[jpopelka]

https://gitlab.cee.redhat.com/dtsd/housekeeping/issues/1546

[miteshvp]

@msrb - any updates here? Have we experienced this again?

[msrb]

@miteshvp yes, we did.

[pkajaba]

We won't be able to resolve this on our own. This problem is caused when worker time is out of sync with Amazon servers [1]. However, we are not able to adjust time inside containers without root privileges.

[1] https://stackoverflow.com/questions/44017410/signature-expired-is-now-earlier-than-error-invalidsignatureexception

EDIT: There might be a remedy for this issue, but Boto does not support it

[2] boto/boto3#1252

[msrb]

Reducing Severity to 2 as per yesterday's post standup discussion.

[msrb]

I haven't seen this error in staging/prod for couple of weeks now. I am going to close this issue. We can reopen it if the same error appears again.