Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

S3 filesystem connection pool depleted when using Rubix Caching #3524

Closed
losipiuk opened this issue Apr 23, 2020 · 14 comments
Closed

S3 filesystem connection pool depleted when using Rubix Caching #3524

losipiuk opened this issue Apr 23, 2020 · 14 comments
Labels
bug Something isn't working

Comments

@losipiuk
Copy link
Member

I exercised Rubix Caching in the same setup as in #3494

After ~20 queries run, the following ones started to failing with io.prestosql.spi.PrestoException: Error reading from s3://<redacted>/tpch-sf1000-ORC/lineitem/20180106_235637_00251_ecxdi_19e8c24a-7fc9-4512-a296-555ab4efc0c2 at position 732097590.

Exception stacktrace from worker log:

2020-04-23T09:34:25.262Z	ERROR	20200423_092732_00026_abrtd.1.0-23-130	com.qubole.rubix.core.CachingInputStream	Failed to read from rubix for file s3://<redacted>/tpch-sf1000-ORC/lineitem/20180106_235637_00251_ecxdi_0dbfd7b1-920d-43b6-8179-8c53dab105d0 position 1331633830 length 7740977. Falling back to remote
java.util.concurrent.ExecutionException: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
	at com.google.shaded.shaded.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
	at com.google.shaded.shaded.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
	at com.google.shaded.shaded.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
	at com.qubole.rubix.core.CachingInputStream.readInternal(CachingInputStream.java:276)
	at com.qubole.rubix.core.CachingInputStream.read(CachingInputStream.java:207)
	at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
	at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:121)
	at org.apache.hadoop.fs.BufferedFSInputStream.readFully(BufferedFSInputStream.java:113)
	at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
	at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:111)
	at io.prestosql.plugin.hive.orc.HdfsOrcDataSource.readInternal(HdfsOrcDataSource.java:62)
	at io.prestosql.orc.AbstractOrcDataSource.readFully(AbstractOrcDataSource.java:96)
	at io.prestosql.orc.AbstractOrcDataSource.readFully(AbstractOrcDataSource.java:87)
	at io.prestosql.orc.AbstractOrcDataSource$LazyBufferLoader.load(AbstractOrcDataSource.java:223)
	at io.prestosql.orc.AbstractOrcDataSource$LazyBufferLoader.loadNestedDiskRangeBuffer(AbstractOrcDataSource.java:210)
	at io.prestosql.orc.AbstractOrcDataSource$MergedOrcDataReader.seekBuffer(AbstractOrcDataSource.java:275)
	at io.prestosql.orc.stream.CompressedOrcChunkLoader.ensureCompressedBytesAvailable(CompressedOrcChunkLoader.java:165)
	at io.prestosql.orc.stream.CompressedOrcChunkLoader.nextChunk(CompressedOrcChunkLoader.java:115)
	at io.prestosql.orc.stream.OrcInputStream.advance(OrcInputStream.java:204)
	at io.prestosql.orc.stream.OrcInputStream.read(OrcInputStream.java:96)
	at io.prestosql.orc.protobuf.CodedInputStream$StreamDecoder.tryRefillBuffer(CodedInputStream.java:2715)
	at io.prestosql.orc.protobuf.CodedInputStream$StreamDecoder.isAtEnd(CodedInputStream.java:2654)
	at io.prestosql.orc.protobuf.CodedInputStream$StreamDecoder.readTag(CodedInputStream.java:2004)
	at io.prestosql.orc.proto.OrcProto$RowIndex.<init>(OrcProto.java:10045)
	at io.prestosql.orc.proto.OrcProto$RowIndex.<init>(OrcProto.java:10017)
	at io.prestosql.orc.proto.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:10710)
	at io.prestosql.orc.proto.OrcProto$RowIndex$1.parsePartialFrom(OrcProto.java:10705)
	at io.prestosql.orc.protobuf.AbstractParser.parseFrom(AbstractParser.java:91)
	at io.prestosql.orc.protobuf.AbstractParser.parseFrom(AbstractParser.java:96)
	at io.prestosql.orc.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
	at io.prestosql.orc.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
	at io.prestosql.orc.proto.OrcProto$RowIndex.parseFrom(OrcProto.java:10243)
	at io.prestosql.orc.metadata.OrcMetadataReader.readRowIndexes(OrcMetadataReader.java:203)
	at io.prestosql.orc.metadata.ExceptionWrappingMetadataReader.readRowIndexes(ExceptionWrappingMetadataReader.java:96)
	at io.prestosql.orc.StripeReader.readColumnIndexes(StripeReader.java:422)
	at io.prestosql.orc.StripeReader.readStripe(StripeReader.java:148)
	at io.prestosql.orc.OrcRecordReader.advanceToNextStripe(OrcRecordReader.java:504)
	at io.prestosql.orc.OrcRecordReader.advanceToNextRowGroup(OrcRecordReader.java:446)
	at io.prestosql.orc.OrcRecordReader.nextPage(OrcRecordReader.java:367)
	at io.prestosql.plugin.hive.orc.OrcPageSource.getNextPage(OrcPageSource.java:99)
	at io.prestosql.plugin.hive.HivePageSource.getNextPage(HivePageSource.java:244)
	at io.prestosql.operator.TableScanOperator.getOutput(TableScanOperator.java:302)
	at io.prestosql.operator.Driver.processInternal(Driver.java:379)
	at io.prestosql.operator.Driver.lambda$processFor$8(Driver.java:283)
	at io.prestosql.operator.Driver.tryWithLock(Driver.java:675)
	at io.prestosql.operator.Driver.processFor(Driver.java:276)
	at io.prestosql.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:1075)
	at io.prestosql.execution.executor.PrioritizedSplitRunner.process(PrioritizedSplitRunner.java:163)
	at io.prestosql.execution.executor.TaskExecutor$TaskRunner.run(TaskExecutor.java:484)
	at io.prestosql.$gen.Presto_332____20200423_081233_2.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1201)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1147)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:796)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:764)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:738)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:698)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:680)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:544)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:524)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5054)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5000)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1486)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.lambda$openStream$2(PrestoS3FileSystem.java:1106)
	at io.prestosql.plugin.hive.util.RetryDriver.run(RetryDriver.java:130)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.openStream(PrestoS3FileSystem.java:1101)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.openStream(PrestoS3FileSystem.java:1086)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.seekStream(PrestoS3FileSystem.java:1079)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.lambda$read$1(PrestoS3FileSystem.java:1023)
	at io.prestosql.plugin.hive.util.RetryDriver.run(RetryDriver.java:130)
	at io.prestosql.plugin.hive.s3.PrestoS3FileSystem$PrestoS3InputStream.read(PrestoS3FileSystem.java:1022)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
	at com.qubole.rubix.core.DirectReadRequestChain.call(DirectReadRequestChain.java:68)
	at com.qubole.rubix.core.DirectReadRequestChain.call(DirectReadRequestChain.java:28)
	at com.google.shaded.shaded.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
	at com.google.shaded.shaded.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
	at com.google.shaded.shaded.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
	... 3 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
	Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
		... 31 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
		Suppressed: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
			... 31 more
		Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
			at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
			at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
			at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
			at java.base/java.lang.reflect.Method.invoke(Method.java:566)
			at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
			at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
			at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
			at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
			at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
			at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
			at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
			at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
			... 29 more
	Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
		at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
		at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
		at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
		at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
		at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
		at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
		at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
		at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
		at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
		... 29 more
Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:314)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
	at jdk.internal.reflect.GeneratedMethodAccessor204.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke(ClientConnectionRequestFactory.java:70)
	at com.amazonaws.http.conn.$Proxy192.get(Unknown Source)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at com.amazonaws.http.apache.client.impl.SdkHttpClient.execute(SdkHttpClient.java:72)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1323)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1139)
	... 29 more

cc: @stagraqubole
@losipiuk losipiuk added the bug Something isn't working label Apr 23, 2020
@shubhamtagra
Copy link
Member

This has been fixed in latest version (0.3.8). We noticed that lot of connections are left with OS in TIME_WAIT state leading to depletion of sockets. It has been fixed by pooling connections in rubix and reusing them. c.c @harmandeeps

@shubhamtagra
Copy link
Member

This has been fixed in latest version (0.3.8). We noticed that lot of connections are left with OS in TIME_WAIT state leading to depletion of sockets. It has been fixed by pooling connections in rubix and reusing them. c.c @harmandeeps

This might not be it, the part that this solves would not apply to embedded mode used here. Will see if I can get this repro to while checking #3494

@shubhamtagra
Copy link
Member

@losipiuk is it possible to repeat your experiment with parallel warmup disabled?

@losipiuk
Copy link
Member Author

@losipiuk is it possible to repeat your experiment with parallel warmup disabled?

Sure I can try to run it. Not sure if this one is easily reproducable though. Just saw it once.

@losipiuk
Copy link
Member Author

FYI: With hive.cache.parallel-warmup-enabled=false I did not reproduce this one.

@shubhamtagra
Copy link
Member

I have trying to repro this but with no luck yet. @losipiuk can you help with few details:

  1. The 20 queries are run in parallel or sequential?
  2. Once the error starts to occur, it stays there for ever or does it resolve itself?
  3. You mentioned 60 cpu limit on worker container, do you know if this directly converts into 59 value for Runtime.getRuntime().availableProcessors()? Trying to check here that if this is really related to rubix or not because with higher value for Runtime.getRuntime().availableProcessors() we would exhaust default s3 connection pool of 500 with 20 queries under specific order of execution.

@losipiuk
Copy link
Member Author

I have trying to repro this but with no luck yet. @losipiuk can you help with few details:

  1. The 20 queries are run in parallel or sequential?

I was running those sequentially.

  1. Once the error starts to occur, it stays there for ever or does it resolve itself?

I do not have the cluster anymore. What I observed is that starting at one of the queries the issues started to occur. And it persisted through rest of the queries which were run (each taking lots of time due to timeouts). I did not observe healthy cluster after issues started to occur.

  1. You mentioned 60 cpu limit on worker container, do you know if this directly converts into 59 value for Runtime.getRuntime().availableProcessors()? Trying to check here that if this is really related to rubix or not because with higher value for Runtime.getRuntime().availableProcessors() we would exhaust default s3 connection pool of 500 with 20 queries under specific order of execution.

I looked at logged configuration at presto startup for task.max-worker-threads which defaults to 2*Runtime.getRuntime().availableProcessors(). It is 120 so Runtime.getRuntime().availableProcessors() was 60 in test env.

@shubhamtagra
Copy link
Member

Ok thanks @losipiuk , I will continue the repro efforts.

@losipiuk losipiuk mentioned this issue Apr 29, 2020
Closed
@shubhamtagra
Copy link
Member

Update on this, I was able to repro it with queries running in a loop with cancellations on every other query. Got into the error state quite early, within half an hour. Tried it over the branch with qubole/rubix#368 as it changes a lot of existing code, I could not repro and same tests ran fine for hours.

@shubhamtagra
Copy link
Member

Found the root cause for this. Explained it in detail in qubole/rubix#375.
It happens due to a race between PrestoS3InputStream.close() due to recordReader closing as part of Driver thread and PrestoS3InputStream.seekStream() happening due to a read in parallel
in rubix. It wasn't happening with the fix for qubole/rubix#368 because with that fix we read very less data from remote and this issue happens only when reading
from remote filesytem. But even with that fix, with enough reads from remote filesystem along with query cancelations we can see this issue.

@shubhamtagra
Copy link
Member

shubhamtagra commented May 6, 2020 •
edited
Loading

Not seeing it with qubole/rubix#368 not just because of lower reads from remote filesystem but because we moved to positional reads in rubix master and this race doesnt happen with positional reads.

@sopel39
Copy link
Member

sopel39 commented May 19, 2020

@stagraqubole does qubole/rubix#368 fix the issue?

@shubhamtagra
Copy link
Member

qubole/rubix@87ae6e1 has fixed it for Presto. This will not occur after rubix upgrade.

@sopel39
Copy link
Member

sopel39 commented May 19, 2020

Fixed by: #3772 (as it upgrades Rubix so that it has qubole/rubix#368).

@sopel39 sopel39 closed this as completed May 19, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

No branches or pull requests
3 participants
@losipiuk @sopel39 @shubhamtagra