Reconnect flow does not check for dead sockets

[george-unbotify]

It seems that reconnect() only checks if socket exists and does not check for disconnected sockets.

To reproduce the issue:

1. Run a python fluentd client and a fluentd service
2. Send data from client to fluentd
3. Restart fluentd while keeping client running
4. Try to send data again to fluentd using same client

Step 4 fails due to a connection reset by peer error.

[arcivanov]

You cannot check if the connection is valid or invalid. Once the socket is determined to back a broken connection by an act of sending data and reading back a response, the socket will be closed, reference set to None and and a new socket will be created on the next reconnect.

[george-unbotify]

You can try to handle connection errors with some sort of a retry.
If not - at least add a documentation stating recommending the user to add a retry mechanism in case the socket is dead.

[arcivanov]

@george-unbotify Retries are not possible without server-based ACK which is a separate issue #120 altogether. Without ACK, there is no possibility, even in theory to know whether the socket is dead unless the router actively sends ICMP to indicate the connection has been broken or kernel detects it by other means. This is a fundamental limitation of the TCP protocol and has been there since 1974.

If the socket errors out, the reconnect is called automatically after the socket reference it's cleared.

Reconnect on every send if necessary:

• https://github.com/fluent/fluent-logger-python/blob/master/fluent/sender.py#L186

Close and reset socket if any error occurred and was detected:

• https://github.com/fluent/fluent-logger-python/blob/master/fluent/sender.py#L158
• https://github.com/fluent/fluent-logger-python/blob/master/fluent/sender.py#L243

[arcivanov]

Furthermore, even in theory, a user cannot add a retry mechanism of any kind with any degree of reliability without server-side ack.