S3 Endpoint URL does not with localstack

[lauzadis]

Hi, we are using Aws SDK Kotlin to handle the AWS operations in our project, but update to the latest AWS SDK Kotlin 0.35-beta, we will face the endpoint url setting as pathstyle or host style, but either use current http://localhost:4566 or http://s3.localhost.localstack.cloud:4566, when running my tests locally I got exception like this.

ws.sdk.kotlin.services.s3.model.S3Exception: Failed to parse response as 'restXml' error
	at aws.sdk.kotlin.services.s3.serde.HeadBucketOperationDeserializerKt.throwHeadBucketError(HeadBucketOperationDeserializer.kt:46)
	at aws.sdk.kotlin.services.s3.serde.HeadBucketOperationDeserializerKt.access$throwHeadBucketError(HeadBucketOperationDeserializer.kt:1)
	at aws.sdk.kotlin.services.s3.serde.HeadBucketOperationDeserializer.deserialize(HeadBucketOperationDeserializer.kt:24)
	at aws.smithy.kotlin.runtime.http.operation.SdkOperationExecutionKt$decorate$3.invoke(SdkOperationExecution.kt:192)
	at aws.smithy.kotlin.runtime.http.operation.SdkOperationExecutionKt$decorate$3.invoke(SdkOperationExecution.kt:192)
...

Before storing files into S3, we use a headbucket to check the existence of bucket name, if it is not existed, created one, but the headbucket code caused the exception.

suspend fun S3Client.bucketExists(s3bucket: String) = try {
    headBucket(HeadBucketRequest { bucket = s3bucket })
    true
} catch (e: Exception) {
    false
}

Any S3Client function will raise an exception, and I can not see any log from the localstack console(I am using a LocalStack v3 from Docker).

Originally posted by @hantsy in #1121

[lauzadis]

Hi @hantsy, I'm not able to replicate this using the latest version of the SDK and LocalStack. I can make S3 requests to my local stack such as ListBuckets and CreateBucket, and then use HeadBucket to determine if a bucket exists or not. Using your code for inspiration, my replication attempt works fine.

Here is my small sample. I used endpointUrl and forcePathStyle always set to true. I tried both endpoint URL options, http://localhost:4566 and http://s3.localhost.localstack.cloud:4566/, they both worked for me.

    val client = S3Client.fromEnvironment {
        credentialsProvider = StaticCredentialsProvider(credentials = Credentials("foo", "bar", "baz"))
        endpointUrl = Url.parse("http://localhost:4566")
        forcePathStyle = true
        region = "us-east-1"
    }

    try {
        client.headBucket {
            bucket = "non-existent-bucket"
        }
    } catch (e: Exception) {
        println("Bucket does not exist")
    }

    println(client.listBuckets{})

I modified this code to also create a bucket, and verified that the HeadBucket operation was successful.

Can you please try my simplistic example to see if it works for you? That would help narrow down the root cause of your issues. Also, double check to make sure your versions are all set to the latest (LocalStack latest, SDK 0.35.0-beta).

Also, can you share what is the latest version of the SDK where this did work for you? It can help us narrow down changes which may have caused this breakage for you.

[hantsy]

I also tried to use endpointUrl and forcePathStyle in my codes.

    @Bean
    fun awsS3Client(awsCredentialsProvider: CredentialsProvider, properties: AwsProperties): S3Client {
        return S3Client {
            credentialsProvider = awsCredentialsProvider
            region = properties.region
//            endpointProvider = properties.endpoint?.let { url ->
//                S3EndpointProvider { Endpoint(url) }
//            } ?: DefaultS3EndpointProvider()
            endpointUrl= properties.endpoint?.let { Url.parse(it) }
            forcePathStyle = !properties.endpoint.isNullOrBlank()
            logMode = LogMode.LogRequestWithBody
        }
    }

It seems the credentialsProvider is a little different.

https://github.com/hantsy/aws-sdk-kotlin-spring-example/blob/master/src/main/kotlin/com/example/demo/internal/AwsConfig.kt#L23-L34

    fun awsCredentialsProvider(properties: AwsProperties): CredentialsProvider {
        val chain = CredentialsProviderChain(
            EnvironmentCredentialsProvider(),
            StaticCredentialsProvider(
                Credentials(
                    secretAccessKey = properties.secretAccessKey,
                    accessKeyId = properties.accessKeyId
                )
            )
        )

        return CachedCredentialsProvider(chain)
    }

[hantsy]

@lauzadis I add Github action workflow file, it build successfully on Github action.

I realize it is my fault. On my local machine, I was always using a VPN proxy client, but not sure now the Proxy settings affect the AWS S3 Client connection.

[lauzadis]

Great, I'm happy you were able to resolve the problem!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.