Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lots of gRPC error output in diag logs #548

Open
shawaj opened this issue Apr 14, 2023 · 2 comments
Open

lots of gRPC error output in diag logs #548

shawaj opened this issue Apr 14, 2023 · 2 comments
Assignees
@MuratUrsavas

Comments

@shawaj
Copy link
Member

shawaj commented Apr 14, 2023

diag logs are getting spammed with a lot of this type of stuff on startup:

INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
INFO:root:Running periodic hardware diagnostics
2023-04-14 03:35:56,160 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(42) - rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
ERROR:hm_pyhelper.miner_param:rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
2023-04-14 03:35:56,161 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(43) - <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 87, in get_pubkey
    encoded_key = self.stub.pubkey(local_pb2.pubkey_req()).address
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
ERROR:hm_pyhelper.miner_param:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 87, in get_pubkey
    encoded_key = self.stub.pubkey(local_pb2.pubkey_req()).address
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:"2023-04-14T03:35:55.955126455+00:00"}"
>
2023-04-14 03:35:56,178 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(42) - rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
ERROR:hm_pyhelper.miner_param:rpc error: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
2023-04-14 03:35:56,180 - [ERROR] - hm_pyhelper.miner_param - (gateway_diagnostics.py).call_grpc -- /opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py:(43) - <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 68, in get_region
    region_id = self.get_region_enum()
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 61, in get_region_enum
    return self.stub.region(local_pb2.region_req()).region
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
ERROR:hm_pyhelper.miner_param:<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
Traceback (most recent call last):
  File "/opt/python-dependencies/hw_diag/diagnostics/gateway_diagnostics.py", line 40, in call_grpc
    return getattr(client, method_name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 68, in get_region
    region_id = self.get_region_enum()
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/hm_pyhelper/gateway_grpc/client.py", line 61, in get_region_enum
    return self.stub.region(local_pb2.region_req()).region
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 1030, in __call__
    return _end_unary_response_blocking(state, call, False, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python-dependencies/grpc/_channel.py", line 910, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused"
	debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:172.17.0.7:4467: Failed to connect to remote host: Connection refused {created_time:"2023-04-14T03:35:56.175259905+00:00", grpc_status:14}"
>
2023-04-14 03:35:56,312 - [INFO] - hm_pyhelper.miner_param - (miner_param.py).run_gateway_mfr -- /opt/python-dependencies/hm_pyhelper/miner_param.py:(38) - gateway_mfr response stdout: b'{\n  "key": "112dvU8U5q7acovjtvnMBynBT6rQZ5RUqiy8BoK95yPuXUhT3tbn",\n  "name": "wobbly-pickle-cheetah"\n}\n'

@MuratUrsavas can we catch these and show a better message rather than spamming the logs?
@shawaj
Copy link
Member Author

shawaj commented Apr 14, 2023

also seen this one:

2023-04-14 03:48:02,468 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_ble_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(103) - Found the following BLE Devices: [{'Address': 'DC:A6:32:6A:36:C0', 'Name': 'nebra-6A36BE.local', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
INFO:hw_diag.utilities.hardware:Found the following BLE Devices: [{'Address': 'DC:A6:32:6A:36:C0', 'Name': 'nebra-6A36BE.local', 'Powered': '1', 'Discoverable': '0', 'Pairable': '0', 'Discovering': '0'}]
2023-04-14 03:48:02,469 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(149) - Retrieving list of LTE device(s)
INFO:hw_diag.utilities.hardware:Retrieving list of LTE device(s)
2023-04-14 03:48:02,475 - [INFO] - hw_diag.utilities.hardware - (hardware.py).get_lte_devices -- /opt/python-dependencies/hw_diag/utilities/hardware.py:(176) - Found the following LTE Devices: []
INFO:hw_diag.utilities.hardware:Found the following LTE Devices: []
INFO:root:Diagnostics shipping not requested, skipping.
INFO:root:Diagnostics complete
INFO:root:Running periodic hardware diagnostics
Exception in thread Thread-164 (_run):
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 975, in run
    self._target(*self._args, **self._kwargs)
  File "src/python/grpcio/grpc/_cython/_cygrpc/thread.pyx.pxi", line 53, in grpc._cython.cygrpc._run_with_context._run
  File "/opt/python-dependencies/grpc/_channel.py", line 1547, in _poll_connectivity
    event = channel.watch_connectivity_state(connectivity,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 510, in grpc._cython.cygrpc.Channel.watch_connectivity_state
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 377, in grpc._cython.cygrpc._watch_connectivity_state
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 385, in grpc._cython.cygrpc._watch_connectivity_state
ValueError: Cannot monitor channel state: Channel closed!
2023-04-14 03:49:02,881 - [INFO] - hm_pyhelper.miner_param

@shawaj
Copy link
Member Author

shawaj commented Apr 15, 2023

https://nebraltd.atlassian.net/browse/NEBD-110

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@MuratUrsavas MuratUrsavas

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shawaj @MuratUrsavas