Memory leak in Solr connections leading to Tomcat crash

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When we got bombarded by openai crawling, Tomcat was spinning up too many open threads and crashing daily.

🕵️ Expected behavior

I expected we could take the traffic.

📜 To Reproduce

See SA logs and convo on Slack.

See these errors in catalina.out

"Connection evictor" #762 daemon prio=5 os_prio=0 tid=0x00007fc2700c9000 nid=0xd79b waiting on condition [0x00007fc248002000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
	at java.lang.Thread.run(Thread.java:748)

And when you take ds-view offline, it is explicitly noting connection evictor issues related to [ds-view], noting there is a memory leak spawning new threads and not closing them.

🖥 Environment Info

Chrome

📚 Version of Software Used

v2.14.3

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

No response

[jordanpadams]

This issue may have reared its head again, and may be related to the latest solrj upgrades:

Additional leak possible in ds-view:

04-Dec-2024 11:53:27.292 SEVERE [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [ds-view] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@17fc1173]) and a value of type [org.eclipse.jetty.util.Pool.MonoEntry] (value [MonoEntry@6ae31bac{IDLE,pooled=RetainableByteBuffer@b7ddc10{DirectByteBuffer@3916fe27[p=0,l=0,c=16384,r=0]={<<<>>>\x00\x018\x01\x04\x00\x00\x00\x01...\x00\x00\x00\x00\x00\x00\x00},r=0}}]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

Here is a discussion on possible resolution upgrading Jetty to fix this leak. https://lists.apache.org/thread/n2rgq5l5jngbnpz8my9flk52zk7zg2xb

[jordanpadams]

Most likely tightly coupled with NASA-PDS/registry-legacy-solr#172

[jordanpadams]

Nightly Tomcat restarts have been instantiated by SAs, and seems to avoiding this now. Not an ideal solution. But it works for now. Closing this.

[jordanpadams]

Still not fixed. It is pretty clear this is a leak in the Solr HTTP2Client. When trying to restart Tomcat:

Caused by: java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1343)
	at java.base/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
	at org.eclipse.jetty.io.ManagedSelector.doStop(ManagedSelector.java:137)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.io.SelectorManager.doStop(SelectorManager.java:280)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.io.ClientConnector.doStop(ClientConnector.java:383)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.http2.client.http.HttpClientTransportOverHTTP2.doStop(HttpClientTransportOverHTTP2.java:106)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.stop(ContainerLifeCycle.java:182)
	at org.eclipse.jetty.util.component.ContainerLifeCycle.doStop(ContainerLifeCycle.java:205)
	at org.eclipse.jetty.client.HttpClient.doStop(HttpClient.java:265)
	at org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:132)
	at org.apache.solr.client.solrj.impl.Http2SolrClient.close(Http2SolrClient.java:296)