Scale back sending rate automatically when timeouts are encountered

[tobiasmcnulty]

Here's the traceback. Note it's in reverse order so you need to read from the bottom up.

Nice-to-have: Timeouts and other exceptions (expected or otherwise) do not cause a build-up of messages in the 'sending' status.

2024-11-10 05:20:22.896 | 2024-11-10 12:20:22,895 smpp.Client.140001014882336 WARNING  timed out |  
-- | -- | --
  |   | 2024-11-10 05:20:22.491 | Press Ctrl-C to quit |  
  |   | 2024-11-10 05:20:22.491 | Waiting up to 2 seconds |  
  |   | 2024-11-10 05:20:22.491 | Sentry is attempting to send 2 pending events |  
  |   | 2024-11-10 05:20:22.390 | smpplib.exceptions.ConnectionError |  
  |   | 2024-11-10 05:20:22.390 | raise exceptions.ConnectionError() |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpplib/client.py", line 217, in send_pdu |  
  |   | 2024-11-10 05:20:22.390 | self.send_pdu(ssm) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpplib/client.py", line 408, in send_message |  
  |   | 2024-11-10 05:20:22.390 | self.send_message( |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/client.py", line 214, in <listcomp> |  
  |   | 2024-11-10 05:20:22.390 | return [ |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/client.py", line 213, in split_and_send_message |  
  |   | 2024-11-10 05:20:22.390 | pdus = self.split_and_send_message(sms["short_message"], **params) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/client.py", line 181, in send_mt_messages |  
  |   | 2024-11-10 05:20:22.390 | self.send_mt_messages() |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/client.py", line 239, in listen |  
  |   | 2024-11-10 05:20:22.390 | client.listen() |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/smpp.py", line 59, in smpplib_main_loop |  
  |   | 2024-11-10 05:20:22.390 | smpplib_main_loop( |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/smpp.py", line 74, in start_smpp_client |  
  |   | 2024-11-10 05:20:22.390 | start_smpp_client(options) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpp_gateway/management/commands/smpp_client.py", line 85, in handle |  
  |   | 2024-11-10 05:20:22.390 | output = self.handle(*args, **options) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 458, in execute |  
  |   | 2024-11-10 05:20:22.390 | self.execute(*args, **cmd_options) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 412, in run_from_argv |  
  |   | 2024-11-10 05:20:22.390 | self.fetch_command(subcommand).run_from_argv(self.argv) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 436, in execute |  
  |   | 2024-11-10 05:20:22.390 | utility.execute() |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line |  
  |   | 2024-11-10 05:20:22.390 | execute_from_command_line(sys.argv) |  
  |   | 2024-11-10 05:20:22.390 | File "/code/manage.py", line 10, in <module> |  
  |   | 2024-11-10 05:20:22.390 | Traceback (most recent call last): |  
  |   | 2024-11-10 05:20:22.390 |   |  
  |   | 2024-11-10 05:20:22.390 | During handling of the above exception, another exception occurred: |  
  |   | 2024-11-10 05:20:22.390 |   |  
  |   | 2024-11-10 05:20:22.390 | TimeoutError: timed out |  
  |   | 2024-11-10 05:20:22.390 | self._socket.sendall(generated) |  
  |   | 2024-11-10 05:20:22.390 | File "/usr/local/lib/python3.10/site-packages/smpplib/client.py", line 214, in send_pdu |  
  |   | 2024-11-10 05:20:22.390 | Traceback (most recent call last): |  
  |   | 2024-11-10 05:20:21.892 | 2024-11-10 12:20:21,891 smpp.Client.140256232824864 WARNING  timed out |  
  |   | 2024-11-10 05:20:04.245 | 2024-11-10 12:20:04,244 smpp.Client.140001014882336 INFO     Entering main listen loop

[tobiasmcnulty]

@simonkagwi Here's another potential issue to pick up. The idea would be to catch the timeout in Python, and instead of allowing the process to exit, decrease the number of messages sent in each batch progressively until a timeout is not encountered anymore, perhaps with a logging warning to let us know this is happening.

[simonkagwi]

@tobiasmcnulty Got it! I'll look into it