Odroid Tornado Issue

[OmerShubi]

#TODO Add Odroid log

[OmerShubi]

cameralog_191003_110815_camera.txt-2019-10-03 11:08:55,857 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:55,879 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:55,894 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.31168611111
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,006 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,029 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,043 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.31974444444
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,157 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,179 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,189 [MainThread  ] [DEBUG]  Worker HB tick, current liveness: -1
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,190 [MainThread  ] [DEBUG]  Worker lost connection
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,191 [MainThread  ] [DEBUG]  Shutdown of the worker
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,192 [MainThread  ] [DEBUG]  Stopping the HB ticker
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,193 [MainThread  ] [DEBUG]  Closing the stream
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,195 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.312475
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,202 [MainThread  ] [ERROR]  Exception in callback <bound method Server._tick of <CameraNetwork.server.Server object at 0xabe98910>>
cameralog_191003_110815_camera.txt-Traceback (most recent call last):
cameralog_191003_110815_camera.txt:  File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 1229, in _run
cameralog_191003_110815_camera.txt-    return self.callback()
cameralog_191003_110815_camera.txt-  File "/home/odroid/code/cameranetwork/CameraNetwork/mdp/worker.py", line 166, in _tick
cameralog_191003_110815_camera.txt-    self._delayed_reconnect.start()
cameralog_191003_110815_camera.txt-  File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 52, in start
cameralog_191003_110815_camera.txt-    self.io_loop.add_timeout(self._next_timeout, self._run)
cameralog_191003_110815_camera.txt-AttributeError: 'DelayedCallback' object has no attribute 'io_loop'
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,214 [MainThread  ] [WARNI]  Got events for closed stream None
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,309 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,329 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,344 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.30869722222
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,456 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,479 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,493 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.31061944444
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,605 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,629 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,644 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.30574722222
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,757 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,779 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,793 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.30375833333
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,906 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,929 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:56,943 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.30348333333
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,056 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,081 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,101 [Thread-1    ] [DEBUG]  Exp.: 186.64, Gain: 0, image range: [0, 255], Value: 1.30125555556
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,110 [Thread-1    ] [DEBUG]  Setting camera to large size
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,259 [Thread-1    ] [DEBUG]  Free up memory
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,260 [Thread-1    ] [DEBUG]  Allocating memory
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,419 [Thread-1    ] [INFO ]  Setting pixel clock to 25.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,427 [Thread-1    ] [DEBUG]  The IDS wrapper does not support automatic gain
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,758 [Thread-1    ] [DEBUG]  Capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,933 [Thread-1    ] [DEBUG]  Finished capturing frame 0.
cameralog_191003_110815_camera.txt-2019-10-03 11:08:57,958 [Thread-1    ] [DEBUG]  Centroid of suns: [ 433.18076722  488.56028664]
cameralog_191003_110815_camera.txt-2019-10-03 11:08:58,005 [MainThread  ] [INFO ]  Measured angle: 72
cameralog_191003_110815_camera.txt-2019-10-03 11:08:58,057 [MainThread  ] [INFO ]  Either failed fitting or not enough measurements
cameralog_191003_110815_camera.txt-2019-10-03 11:08:58,057 [MainThread  ] [INFO ]  Using measured angle: 72
cameralog_191003_110815_camera.txt-2019-10-03 11:08:58,059 [MainThread  ] [DEBUG]  Finished procesing cmd: sunshader_update

[Addalin]

If for some reason the communication between the proxy and the camera fails, and they are both working, automat reconnection doesn't work.
To Do : add logging in "/home/odroid/code/cameranetwork/CameraNetwork/mdp/worker.py" to debug this issue.
check this function :
self._delayed_reconnect.start()