Missing Bucket in path when using "forcePathStyle"

[clementguillot]

Describe the bug

Using a local develoment environment with any S3-comptatible backend (e.g. minio), I try to configure a S3 Client with forcePathStyle = true.

When I call .presignGetObject, the result URL does not contain the bucket in the path. However, it works as expected when this parameter is false or null.

Expected behavior

When forcePathStyle is false or null, presigned URL has pattern http://{bucket}.{endpoint}/{key}.

But, once this parameter is true, presigned URL has pattern http://{endpoint}/{key}and bucket is missing, expected URL is http://{endpoint}/{bucket}/{key}.

Current behavior

Bucket is missing from built-URL. When accessing the generated URL, server throws a 403 error.

Steps to Reproduce

Let's consider the following sippnet:

val localS3 =
  S3Client {
    endpointUrl = Url.parse("http://localhost:9000")
    region = "us-east-1"
    forcePathStyle = true
    credentialsProvider =
      StaticCredentialsProvider {
        accessKeyId = "access-key"
        secretAccessKey = "secret-key"
      }
  }

val getRequest =
  GetObjectRequest {
    bucket = "my-bucket"
    key = "my-file-path"
  }

val url = localS3.presignGetObject(getRequest, 1.hours).url.toString()

url looks like this: http://localhost:9000/my-file-path?x-id=GetObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...
Expected: http://localhost:9000/my-bucket/my-file-path?x-id=GetObject&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=...

Note that with the same local S3 backend, but using the following NodeJS snippet, it works as expected:

const s3client = new S3Client({
  endpoint: "http://localhost:9000",
  region: "us-east-1",
  credentials: {
    accessKeyId: "access-key",
    secretAccessKey: "secret-key",
  },
  forcePathStyle: true,
})

const getCommand = new GetObjectCommand({
  Bucket: "my-bucket",
  Key: "my-file-path"
})

getSignedUrl(s3client, getCommand, {expiresIn: 3600}).then((result) => {
  console.log("pre-signed URL" + result)
})

Print: http://localhost:9000/my-bucket/my-file-path?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=...

Possible Solution

No response

Context

No response

AWS Kotlin SDK version used

1.0.30

Platform (JVM/JS/Native)

OpenJDK Runtime Environment GraalVM CE 17.0.7+7.1 and Kotlin 1.9.22

Operating System and version

Linux Ubuntu 22.04.3

[ianbotsf]

Hi @clementguillot, thanks for the bug report. I've been able to reproduce the issue locally. It appears to be related to presigning and not necessarily to using a custom endpoint (e.g., for MinIO). I'll root cause the issue and get a fix posted shortly.

[ianbotsf]

The fix for this issue has been merged to main and should be included in tomorrow's release. Closing this for now but please re-open if the issue still occurs on 1.0.32 or later.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[clementguillot]

Thank you @ianbotsf for the fix!

[clementguillot]

@ianbotsf unfortunatelly, using v1.0.32, bucket's name is now part of the presigned URL by the link doesn't work and Minio throws a 403:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
    <Key>65983dbe7ee2fa0039df4ff1/1689923702657342873</Key>
    <BucketName>my-bucket</BucketName>
    <Resource>/my-bucket/my-file/path</Resource>
    <RequestId>17A93FCF1CFEE650</RequestId>
    <HostId>dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8</HostId>
</Error>

[ianbotsf]

Hmm, that's unexpected. We added a new presigning E2E which uses forcePathStyle = true and it passes when run against S3. Can you reproduce the failure on S3 directly?

[clementguillot]

Hi @ianbotsf,

I tried a couple of scenarios as below:

• Using S3, it works as expected with path style. URL looks like this: https://s3.eu-west-3.amazonaws.com/test-bucket-cguillot/my-file...... and both GET and PUT are ok.
• Using Scaleway (French provider), also works as expected.
• Using local Minio in Docker:
  • Works as expected with NodeJS, server returns the file or 404 if the key doesn't exist
  • Works as expected with Rush, same behavior as Node
  • Doesn't work with Kotlin SDK, backend returns 403 error with SignatureDoesNotMatch

Unfortunately, I can't say if the error is on Minio's side, but it seems that Node and Rust SDKs can tackle this. It would be definitely great if Kotlin SDK could too ;)

Please let me know if there is anything I can do to help with this!

EDIT: I remember that we had a "hack" in our Node codebase for a while due to that issue, when using a specific port (:9000) in the endpoint: aws/aws-sdk-js-v3#2121 maybe could it be related?

[clementguillot]

@ianbotsf I confirm that Kotlin SDK works as expected when local Minio is on port 80 (or 443). Issue is no longer related to path style but is about using a port in the endpoint URL. Maybe should I open another issue?

[ianbotsf]

Yes this should be a new issue. I've opened #1177 and I'll re-close this one.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.