S3 head_object fails if called directly after instanciating a client

[pfrayer]

Describe the bug

• Calling boto3's head_object just after instantiating a boto3.client fails with 400 - Bad Request.
• Calling boto3's head_object after calling any other method (eg. get_object, list_objects etc.) works.

Expected Behavior

Calling boto3's head_object just after instantiating a boto3.client should works, returns HTTP 200 & the related object's metadata.

Current Behavior

botocore.exceptions.ClientError is raised (see reproduction steps)

Reproduction Steps

Run a Python 3.12 bare image

$ docker run --rm -it python:3.12 bash

Once inside the image:

# Install latest boto3
$ pip install boto3
...
Successfully installed boto3-1.35.54 botocore-1.35.54 jmespath-1.0.1 python-dateutil-2.9.0.post0 s3transfer-0.10.3 six-1.16.0 urllib3-2.2.3

# Run Python. Instantiate a boto3 client, try to HEAD an object in S3
$ python
Python 3.12.6 (main, Sep 12 2024, 21:12:04) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3_client = boto3.client( # Credentials here are OK
...     "s3",
...     endpoint_url=S3_AUTH_URL,
...     aws_access_key_id=S3_AUTH_ACCESS_KEY,
...     aws_secret_access_key=S3_AUTH_SECRET_KEY
... )
>>> s3_client.head_object(
...     Bucket="existing_bucket",
...     Key="existing_object"
... )
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (400) when calling the HeadObject operation: Bad Request

Here I can retry s3_client.head_object() as many time as I want: it will always fail with 400 - Bad Request.

Now, restart the process from the point where we start Python, and do a get_object (or any other call like list_objects) before calling head_object: head_object will works.

$ python
Python 3.12.6 (main, Sep 12 2024, 21:12:04) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3_client = boto3.client( # Credentials here are OK
...     "s3",
...     endpoint_url=S3_AUTH_URL,
...     aws_access_key_id=S3_AUTH_ACCESS_KEY,
...     aws_secret_access_key=S3_AUTH_SECRET_KEY
... )
>>> s3_client.get_object(  # This works
...     Bucket="existing_bucket",
...     Key="existing_object"
... )
{'ResponseMetadata': {'HTTPStatusCode': 200,... }
>>> s3_client.head_object(  # Now, this works too
...     Bucket="existing_bucket",
...     Key="existing_object"
... )
{'ResponseMetadata': {'HTTPStatusCode': 200,... }

Possible Solution

head_object works directly after instantiating the client only if region_name was given in the client's params

>>> import boto3
>>> s3_client = boto3.client( # Credentials here are OK
...     "s3",
...     endpoint_url=S3_AUTH_URL,
...     aws_access_key_id=S3_AUTH_ACCESS_KEY,
...     aws_secret_access_key=S3_AUTH_SECRET_KEY,
...    region_name=S3_REGION
... )
>>> s3_client.head_object(  # Now, this works too
...     Bucket="existing_bucket",
...     Key="existing_object"
... )
{'ResponseMetadata': {'HTTPStatusCode': 200,... }

Additional Information/Context

No response

SDK version used

1.35.51,1.35.54

Environment details (OS name and version, etc.)

Debian 12

[adev-code]

Hello @pfrayer, thanks for reaching out. I have replicated the same setup and I was not able to reproduce the same issue. Could you provide the full debug logs of the behavior? You can get the full debug logs by adding boto3.set_stream_logger('') and redacting any sensitive information. Thank you.

[adev-code]

Hello @pfrayer, just in addition to my response, please ensure that you're using AWS S3 Endpoint or that support AWS (you could let us know what you are using and the full debug logs). Thank you.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.