upload_fileobj hangs indefinitely without returning error #4148
Comments
fitnycdigitalinitiatives
added
bug
tim-finnigan
added
the
investigating
|
Thanks for reaching out. In the issue you linked someone mentioned that changing the base Docker image fixed this. Have you tried reproducing the issue in environments other than RHEL? Also the latest Boto3 is 1.34.118 per the CHANGELOG, have you tried updating to that version? And are you using the CRT and if so have you tried uninstalling/reinstalling? You could also try testing with the various S3 transfer configuration options documented here, for example: import boto3
from boto3.s3.transfer import TransferConfig
# Create an S3 client
s3 = boto3.client('s3')
# Define the transfer config
transfer_config = TransferConfig(
multipart_threshold=1024 * 25, # 25 MB
max_concurrency=10,
multipart_chunksize=1024 * 1024, # 1 MB
num_download_attempts=5,
max_io_queue=100,
io_chunksize=1024 * 1024, # 1 MB
use_threads=True
)
# Open the local file
with open('file.txt', 'rb') as f:
# Use the transfer config in an S3 upload operation
s3.upload_fileobj(
Fileobj=f,
Bucket='test-bucket',
Key='remote_file.txt',
Config=transfer_config
)Otherwise I recommend following the S3 performance guideline optimizations documented here: https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance-guidelines.html If you're able to narrow down the conditions in which this issue occurs, please let us know. |
tim-finnigan
added
response-requested
|
Hello, Not sure if this is related, but we recently got this error:
|
|
More detailed log: |
github-actions
bot
removed
the
response-requested
|
@tim-finnigan Hello, I did some more digging into the system logs and saw this message popping up around these issues: Do you think that could be causing the problem? |
|
Thanks for following up, I'm not sure that could be causing the problem. Could you provide the full snippet you're using and complete debug logs (with sensitive info redacted) for us to investigate further? Adding |
tim-finnigan
added
the
response-requested
|
Hi Tim, I don't want to speak too soon, but one thing that I recently tried was turning off threads in the TransferConfig and I haven't run into the error in the last several days. Perhaps there's something interrupting the threading process either with our application or in RHEL 9. |
github-actions
bot
removed
the
response-requested
tim-finnigan
added
the
response-requested
|
Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one. |
github-actions
bot
added
the
closing-soon
|
Verifying that turning off threading fixed the issue. I did see online that RHEL 9 changed something with it's threading approach, but it's way beyond my understanding to know if it could be related to what I'm experiencing. |
github-actions
bot
removed
closing-soon
|
I experienced the same issue, ATM using s3_client.put_object method instead. |
|
Thanks for following up, I think we're still waiting on the full debug logs for further investigation of this issue. |
tim-finnigan
added
the
response-requested
|
Checking in — I was made aware of this issue involving gevent: gevent/gevent#1826, which is present in the logs shared earlier here. This issue appears to be specific to RHEL-based systems. Please note that we do not provide or officially support gevent with our networking setup. Any issues related to gevent will need to be addressed by the gevent team. |
tim-finnigan
added
closing-soon
|
Cool, good catch. thanks for your help! |
|
This issue is now closed. Comments on closed issues are hard for our team to see. |
Describe the bug
Hello,
We've recently run into a problem that seems a bit similar to #3657. Occasionally our application will stall out uploading objects to our bucket seemingly randomly using
upload_fileobjwithout returning an error. We've run this application for years without experience this issue, but recent changes have included upgrading boto3 to 1.33.2 and RHEL to 9 (not sure if relevant to the issue). As suggested in the related issue, we turned on the debug logs and here's an example when the upload just completely hangs for hours on end:As you can see it just hangs after the initial PUT. In successful uploads after the first PUT, there's
DEBUG 2024-05-28 19:33:05 s3transfer.utils:utils:release:645: Releasing acquire 0/NoneThere's nothing different about the files that it's failing on vs. the files it uploads without issue. The vast majority of uploads are successful. Just these occasional uploads stall without error.
Expected Behavior
Uploads should fail with an error.
Current Behavior
See above.
Reproduction Steps
Not able to reproduce, happens randomly when using upload_fileobj. The application uses this method:
Possible Solution
No response
Additional Information/Context
No response
SDK version used
boto3 1.33.2
Environment details (OS name and version, etc.)
rhel 9
The text was updated successfully, but these errors were encountered: