An S3Exception caused by an S3 301 HTTP status doesn't actually contain the redirect URL

[changlinli]

Describe the feature

For a 301 HTTP response, actually capture the redirect URL somewhere in an S3Exception to make debugging considerably easier. It's especially annoying with when the following error message is returned

Please send all future requests to this endpoint.

but "this endpoint" is never specified anywhere and can't be found anywhere in the S3Exception instance.

Is your Feature Request related to a problem?

If a 301 status code is returned from S3 we get an S3Exception, but the exception is not very helpful. In particular, while the 301 status indicates that a URL has moved permanently and should specify where the URL has moved to, the corresponding S3Exception does not describe where the URL has moved to anywhere that I could find, which makes both automated error reporting and manual debugging very frustrating.

This is the default toString:

software.amazon.awssdk.services.s3.model.S3Exception: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. (Service: S3, Status Code: 301, Request ID: SOME_REQUEST_ID, Extended Request ID: SOME_LONG_EXTENDED_REQUEST_ID)

If I look at S3Exception.awsErrorDetails() there are two promising methods rawResponse and sdkHttpResponse. Unfortunately rawResponse is null and while sdkHttpResponse does return a SdkHttpFullResponse (after downcasting from SdkHttpResponse), the underlying InputStream of its content is already closed by the time the S3Exception gets thrown and can't be read.

So effectively, AFAICT, nowhere in an S3Exception is the actual redirect URL captured.

Compare this with the raw XML returned by an endpoint, which actually tells us what "this endpoint" is:

<Error>
  <Code>PermanentRedirect</Code>
  <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
  <!-- I want this data below somewhere in an S3Exception! -->
  <Endpoint>$SOME_BUCKET.s3-us-east-1.amazonaws.com</Endpoint>
  <Bucket>$SOME_BUCKET</Bucket>
  <RequestId>$SOME_REQUEST_ID</RequestId>
  <HostId>$SOME_HOST_ID</HostId>
</Error>

Proposed Solution

One way to implement this without changing any of the public API surface to actually pass in non-null documentBytes to software.amazon.awssdk.protocols.query.internal.unmarshall.AwsXmlErrorUnmarshaller.unmarshall, which at least will get us the raw bytes of the HTTP response.

Right now when the XML is being unmarshalled it just gets saddled with an Optional.empty() which eventually turns into a null (see here: https://github.com/aws/aws-sdk-java-v2/blob/5d39aa01fc69269a316cce10667af2564fb42848/core/protocols/aws-xml-protocol/src/main/java/software/amazon/awssdk/protocols/xml/internal/unmarshall/AwsXmlErrorTransformer.java#L51).

We could instead do a similar thing with what is done here https://github.com/aws/aws-sdk-java-v2/blob/5d39aa01fc69269a316cce10667af2564fb42848/core/protocols/aws-query-protocol/src/main/java/software/amazon/awssdk/protocols/query/unmarshall/AwsXmlErrorProtocolUnmarshaller.java#L99 that actually gives us a non-null documentBody.

Describe alternatives you've considered

Could have an actual change to an S3Exception's surface API, but that seems much more invasive.

Acknowledge

• I may be able to implement this feature request

AWS Java SDK version used

2.17.133

JDK version used

openjdk version "16" 2021-03-16

Operating System and version

NixOS 21.05

[debora-ito]

@changlinli thank you for reaching out and for submitting the PR.

Before reviewing the PR I'll investigate if this is something we can fix by changing the S3 JSON model which the exception is based on.

[rhowe]

Is it available from the response headers?

.sdkHttpResponse().firstMatchingHeader("Location")

[debora-ito]

@changlinli I apologize for the lack of response in this issue.

After discussing the suggested change with the team, we decided not to merge the customization on the SDK side. Ideally we'd like to see the change on the S3 model. I'll raise this feature to the S3 team internally and will post any updates we get.

In the meantime, I'll move this to the central aws-sdk for visibility to the other SDKs.

[vgjenks]

I thought I'd throw my $0.02 in here. I'm having the same problem w/ the v3 JS SDK. An exception never actually gets thrown when doing an HTTP PUT on a bucket with a signed URL, but when I test in Postman, outside of the code, I see this:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>PermanentRedirect</Code>
    <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message>
    <Endpoint>s3.amazonaws.com</Endpoint>
    <Bucket>my-bucket-name</Bucket>
    <RequestId>XXXXXXXXXXXXXXXX</RequestId>
    <HostId>MY-HOSTID-VALUE</HostId>
</Error>

It's not very helpful and I have no idea why this is happening.

[kellertk]

P126359672

[steveloughran]

@vgjenks

It's not very helpful and I have no idea why this is happening.
newer regions (milan) are changing how requests against s3 central are redirected. instead of a 301/302 you now have to parse the response

[tim-finnigan]

Thanks for your patience here. The request to support this is continuing to be tracked in the S3 team's backlog. Please refer to the blog or SDK changelog for updates.

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.