No messages received for 16 minutes following ServiceBus TimeoutException #395
Comments
|
Has anybody figured out the issue yet? I am working on a project that seems to have this exact same issue. To me if feels like this is an issue between the Azure IaaS VM Virtual NIC => outbound; we have performed multiple tcpdumps that seem to indicate this. The very random 16 minute requests seem to go out, but nothing comes back until 16 minutes later. This happens maybe 1% of the time, so is unpredictable other than we know it will happen at least a time or two during the day - but usually about 10 to 20 times per day (out of thousands of requests). We actually set up some automated requests to test whether or not there were patterns, and to help us identify times of the long-running requests. It appears there is no pattern. Here is a sample of data I collected: |
|
We switched to version 1.2.18 and the problem appears to be fixed, When this happens now we see the connection get aborted after 30 seconds, it then reconnects and resumes processing. This all happens inside the azure client code and we made no changes to our code that calls it. |
|
Thanks for the info! I will pass this on. We discovered the same issue that exists with private link does not seem to exist if instead we use Vnet peering. |
Actual Behavior
Summary of problem
Some UpdateState requests fail with TimeoutException and we then don't receive any messages from the queue for 16 minutes when the connection times out and reconnects. If this is expected behaviour is there a way we can configure a much shorter timeout?
Details with lots of log output
We process messages in a loop until we we stop our process or we get an exception. We have set prefetch=10 and if the receive() call returns no messages we sleep for 2 seconds. Strip away all the error handling, etc and the loop is effectively calling two IMessageReceiver methods.
Most of the time this works without a restart for weeks at a time but occasionally we get a 16 minute period (it's always 16 minutes) where we receive no messages or failures from the above methods. After the 16 minute delay we start getting messages again and immediately process the backlog of messages so we know messages are being published during this 16 minute periods. Recently we have started getting more of these delays where it can be as often as once a day.
During the 16 minute period the calls to messageReceiver.receive() continue to work and the log messages are the same as when there are no messages on the queue.
The first sign of problem are 8 exceptions indicating timeouts trying to update the message state. When azure recovers 16 minutes later these are the first message we receive. Each of the exceptions are the same, e.g.
For the next 11 minutes all we see logged are the 'No messages received' messages every 12 seconds.
Then after 11 minutes we see the following request.
And 10 seconds later we get this timed out exception.
These requests and exceptions then continue for another 5 minutes until the connection finally times out and we see the following logged.
And sure enough 1 second later it reconnects and we start receiving messages again.
Expected Behavior
Versions
The text was updated successfully, but these errors were encountered: