Fix watchdog thread for umount may hang

[polyrabbit]

We still observe unkillable juicefs processes in our environment, here is one scenario:

1. doUmount call has finished, but it failed.
2. watch dog thread for doUmount stucks in FlushAll.

The watch dog thread is used to monitor doUmount thread, but I think a reverse watch mechanism also needed.

[davies]

The above lock is suspicious, have you find out the cause?

[davies]

FlushAll may take long than 30 seconds, some data may lose if we exit before that.

[polyrabbit]

How about only os.Exit when doUmount fails?

FlushAll only flushes memory buffers. Those are requests not acked to client, I suppose it's OK to drop them under abnormal conditions. Juicefs only promises a close-to-open consistency.

[polyrabbit]

The above lock is suspicious, have you find out the cause?

I suppose it's because user sends too many write requests - there are more than 500 commitThread threads. User's process becomes unresponsive and unkillable ([rclone] <defunct>), so as the juicefs process. I suppose juicefs needs a way to protect itself from such a large write concurrency.