Loop between epoll_wait and accept4

[TETYYS]

When accept4 returns -1, eventcore doesn't check any errno value, and when it returns errno 24 (too many files), an infinite loop occurs.

A loop somewhere between epoll_wait and accept4 occurs when trying to connect to vibe.d webpage when worker then attempts lots of connections to other servers:

#0  0x00007f02e7476ff7 in accept4 (fd=21, addr=..., addr_len=0x7ffc4d279388,    flags=526336) at ../sysdeps/unix/sysv/linux/accept4.c:40
#1  0x000056425a54e756 in _D9eventcore7drivers5posix7sockets__T23PosixEventDriverSocketsHTCQClQCeQBz5epoll14EpollEventLoopZQCi8onAcceptMFNbNfSQEk6driver__T6HandleVAyaa2_6664TmVmN1ZQzZ9__lambda2MFNbNiNeZv (this=0x7ffc4d279400)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/sockets.d:244
#2  0x000056425a54e551 in _D9eventcore7drivers5posix7sockets__T23PosixEventDriverSocketsHTCQClQCeQBz5epoll14EpollEventLoopZQCi8onAcceptMFNbNfSQEk6driver__T6HandleVAyaa2_6664TmVmN1ZQzZv (this=0x7f02e966f000, listenfd=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/sockets.d:244
#3  0x000056425a5605fa in _D9eventcore7drivers5posix6driver14PosixEventLoop__T6notifyVEQChQCaQBvQBs9EventTypei0ZQBiMFNbNfSQDqQCv__T6HandleVAyaa2_6664TmVmN1ZQzZv (    this=0x7f02e966d080, fd=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/driver.d:230
#4  0x000056425a54c43a in eventcore.drivers.posix.epoll.EpollEventLoop.doProcessEvents(core.time.Duration) (this=0x7f02e966d080, timeout=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/epoll.d:55
#5  0x000056425a54bf20 in _D9eventcore7drivers5posix6driver__T20PosixEventDriverCoreHTCQChQCaQBv5epoll14EpollEventLoopHTCQDpQDi5timer22LoopTimeoutTimerDriverHTCQFcQEvQEq6events__T22PosixEventDriverEventsHTQEpHTCQHbQGuQGp7sockets__T23PosixEventDriverSocketsHTQGqZQBfZQDjZQIa13processEventsMFNbNfS4core4time8DurationZEQLcQKh10ExitReason (    this=0x7f02e9670040, timeout=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/driver.d:141
#6  0x000056425a525eeb in vibe.core.task.TaskScheduler.process() (this=...)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/task.d:675
#7  0x000056425a525fcd in vibe.core.task.TaskScheduler.waitAndProcess() (this=...)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/task.d:713
#8  0x000056425a50a76c in _D4vibe4coreQf12runEventLoopFNbNfZi ()    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/core.d:218
#9  0x000056425a50a5f2 in _D4vibe4coreQf14runApplicationFNfPAAyaZi (args_out=0x0)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/core.d:111
#10 0x000056425a289c44 in D main () at source/app.d:65

#0  0x00007f02e74760f3 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1  0x000056425a54c378 in eventcore.drivers.posix.epoll.EpollEventLoop.doProcessEvents(core.time.Duration) (this=0x7f02e966d080, timeout=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/epoll.d:47
#2  0x000056425a54bf20 in _D9eventcore7drivers5posix6driver__T20PosixEventDriverCoreHTCQChQCaQBv5epoll14EpollEventLoopHTCQDpQDi5timer22LoopTimeoutTimerDriverHTCQFcQEvQEq6events__T22PosixEventDriverEventsHTQEpHTCQHbQGuQGp7sockets__T23PosixEventDriverSocketsHTQGqZQBfZQDjZQIa13processEventsMFNbNfS4core4time8DurationZEQLcQKh10ExitReason (    this=0x7f02e9670040, timeout=...)    at ../../.dub/packages/eventcore-0.8.27/eventcore/source/eventcore/drivers/posix/driver.d:141
#3  0x000056425a525eeb in vibe.core.task.TaskScheduler.process() (this=...)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/task.d:675
#4  0x000056425a525fcd in vibe.core.task.TaskScheduler.waitAndProcess() (this=...)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/task.d:713
#5  0x000056425a50a76c in _D4vibe4coreQf12runEventLoopFNbNfZi ()    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/core.d:218
#6  0x000056425a50a5f2 in _D4vibe4coreQf14runApplicationFNfPAAyaZi (args_out=0x0)    at ../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/core.d:111
#7  0x000056425a289c44 in D main () at source/app.d:65

Worker thread function normally. Connections to webpage from client hang going from Established state to FIN wait 2. Single connection may exit loop after a long time (timeout seems random, connection can terminate on client side, but still left hanging on the server) and throw this on vibe.d server:

[main(f5+f) dbg] Accept TLS connection: server
[main(f5+f) ERR] HTTP connection handler has thrown: Accepting SSL tunnel:  (0)
[main(f5+f) dbg] Full error: object.Exception@../../.dub/packages/vibe-d-0.8.3-alpha.4/vibe-d/tls/vibe/stream/openssl.d(378): Accepting SSL tunnel:  (0)
[main(f5+f) dbg] ----------------
[main(f5+f) dbg] ??:? @safe int vibe.stream.openssl.OpenSSLStream.enforceSSL(int, immutable(char)[]) [0x5a4ee504]
[main(f5+f) dbg] ??:? @safe int vibe.stream.openssl.OpenSSLStream.checkSSLRet(int, immutable(char)[]) [0x5a4edff0]
[main(f5+f) dbg] ??:? @safe vibe.stream.openssl.OpenSSLStream vibe.stream.openssl.OpenSSLStream.__ctor(vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress, immutable(char)[][]) [0x5a4ed0f0]
[main(f5+f) dbg] ??:? @safe vibe.stream.openssl.OpenSSLStream vibe.internal.freelistref.internalEmplace!(vibe.stream.openssl.OpenSSLStream, vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress).internalEmplace(void[], ref vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, ref vibe.stream.openssl.OpenSSLContext, ref vibe.stream.tls.TLSStreamState, ref immutable(char)[], ref vibe.core.net.NetworkAddress) [0x5a4bdb26]
[main(f5+f) dbg] ??:? @safe vibe.stream.openssl.OpenSSLStream vibe.internal.freelistref.FreeListObjectAlloc!(vibe.stream.openssl.OpenSSLStream, true, true, int).FreeListObjectAlloc.alloc!(vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress).alloc(vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress) [0x5a4bd877]
[main(f5+f) dbg] ??:? @safe vibe.internal.freelistref.FreeListRef!(vibe.stream.openssl.OpenSSLStream, true).FreeListRef vibe.internal.freelistref.FreeListRef!(vibe.stream.openssl.OpenSSLStream, true).FreeListRef.opCall!(vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress).opCall(vibe.internal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.openssl.OpenSSLContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress) [0x5a4bd697]
[main(f5+f) dbg] ??:? @safe vibe.internal.freelistref.FreeListRef!(vibe.stream.openssl.OpenSSLStream, true).FreeListRef vibe.stream.tls.createTLSStreamFL!(vibe.internal.interfaceernal.interfaceproxy.InterfaceProxy!(vibe.core.stream.Stream).InterfaceProxy, vibe.stream.tls.TLSContext, vibe.stream.tls.TLSStreamState, immutable(char)[], vibe.core.net.NetworkAddress) [0x5a4b9b90]
[main(f5+f) dbg] ??:? @safe void vibe.http.server.handleHTTPConnection(vibe.core.net.TCPConnection, vibe.http.server.HTTPServerContext) [0x5a499cbf]
[main(f5+f) dbg] ??:? nothrow @safe void vibe.http.server.listenHTTPPlain(vibe.http.server.HTTPServerSettings, void delegate(vibe.http.server.HTTPServerRequest, vibe.http.server.HTTPServerResponse) @safe).doListen(vibe.http.server.HTTPServerContext, bool, bool).__lambda4(vibe.core.net.TCPConnection) [0x5a46b861]
[main(f5+f) dbg] ??:? void vibe.core.task.TaskFuncInfo.set!(void delegate(vibe.core.net.TCPConnection) @safe, vibe.core.net.TCPConnection).set(ref void delegate(vibe.core.net.TCPConnection) @safe, ref vibe.core.net.TCPConnection).callDelegate(ref vibe.core.task.TaskFuncInfo) [0x5a5442bc]
[main(f5+f) dbg] ??:? void vibe.core.task.TaskFuncInfo.call() [0x5a525d15]
[main(f5+f) dbg] ??:? nothrow void vibe.core.task.TaskFiber.run() [0x5a52532d]
[main(f5+f) dbg] ??:? void core.thread.Fiber.run() [0xe868f65b]
[main(f5+f) dbg] ??:? fiber_entryPoint [0xe868ea26]
[main(f5+f) dbg] ??:? [0xffffffff]

While at least one connection is hung, main thread always uses 100% CPU (about 55% usermode, 45% kernelmode).

Another symptom when I noticed when the server was malfunctioning is that vibe.d server couldn't serve static files (404 error):

[main(Oumu) dbg] Failed to open file public/css/dataTables.bootstrap.css: object.Exception@../../.dub/packages/vibe-core-1.4.0-alpha.1/vibe-core/source/vibe/core/file.d(41): Failed to open file 'public/css/dataTables.bootstrap.css'

[kubo39]

I think this not an infinite loop, it is a busy loop.
When accept/accept4 returns EMFILE or ENFILE, it does not discard the pending request and readable in epoll_wait.
Possibly solution in userland is sleep +10ms after seeing an EMFILE return code as Facebook says.