feat: support default checksums #1191

0marperez · 2024-11-27T10:53:55Z

Issue #

Description of changes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Signed-off-by: 0marperez <[email protected]>

…flexible-checksums

0marperez · 2024-11-27T16:01:36Z

Note to reviewers: I'll address the breaking API changes and failing protocol issues soon. In the meantime, I’d appreciate a review of the checksum-related changes.

lauzadis · 2024-11-27T19:19:09Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+ * Calculates a request's checksum.
 *
- * If the checksum will be sent as a header, calculate the checksum.
+ * If a user supplies a checksum via an HTTP header no calculation will be done. The exception is MD5, if a user


correctness/clarification: "Calculates a request's checksum" and "If a user supplies a checksum via an HTTP header no calculation will be done." conflict with each other

lauzadis · 2024-11-27T19:21:12Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+ *
+ * @param requestChecksumRequired Model sourced flag indicating if checksum calculation is mandatory.
+ * @param requestChecksumCalculation Configuration option that determines when checksum calculation should be done.
+ * @param userSelectedChecksumAlgorithm The checksum algorithm that the user selected for the request, may be null.


naming: requestChecksumAlgorithm

lauzadis · 2024-11-27T19:21:55Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+    private val forcedToCalculateChecksum = requestChecksumRequired || requestChecksumCalculation == HttpChecksumConfigOption.WHEN_SUPPORTED
+    private val checksumHeader = StringBuilder("x-amz-checksum-")
+    private val defaultChecksumAlgorithm = lazy { Crc32() }
+    private val defaultChecksumAlgorithmHeaderPostfix = "crc32"


Postfix -> Suffix

lauzadis · 2024-11-27T19:24:30Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+     * The header must start with "x-amz-checksum-" followed by the checksum algorithm's name.
+     * MD5 is not considered a valid checksum algorithm.
+     */
+    private fun userProviderChecksumHeader(request: HttpRequest, logger: Logger): String? {


style: can be restructured as an extension function HttpRequest.checksumHeader(logger: Logger): String?

and naming: userProvidedChecksumHeader

lauzadis · 2024-11-27T19:30:00Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

 * Users can check which checksum was validated by referencing the `ResponseChecksumValidated` execution context variable.
 *
- * @param shouldValidateResponseChecksumInitializer A function which uses the input [I] to return whether response checksum validation should occur
+ * @param responseValidationRequired Flag indicating if the checksum validation is mandatory.


docs: "Model sourced flag"

lauzadis · 2024-11-27T19:40:29Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

+            .removePrefix("x-amz-checksum-")
+            .toHashFunction() ?: throw ClientException("Could not parse checksum algorithm from header $checksumHeader")
+
+        if (context.protocolResponse.body is HttpBody.Bytes) {


Why was this branch added? The spec says we should delay checksum calculation until the body is consumed:

Where possible, SDKs MUST defer this calculation and validation until the payload is actually consumed by the user i.e. a payload must not be read twice.

toHashingBody doesn't support Bytes bodies. I decided to calculate the checksum in memory instead of adding support for Bytes to toHashingBody. It should be safe if the request is not being streamed but I think you're right that we're not following the spec

toHashingBody doesn't support Bytes bodies

Does it need to? The previous implementation never seemed to need it

I think it does, the spec has some unit tests with non-streaming bodies

Discussed offline and we'll be removing support for HttpBody.Bytes from FlexibleChecksumsResponseInterceptor because HTTP bodies are never bytes. Source

I think removing support for HttpBody.Bytes is a bad idea. Our current implementations of OkHttp and CRT engines do not return bytes, that's true, but future implementations or other engines wrapped by users may. In addition, interceptors may be used to rewrite response bodies by streaming them into memory, processing them, and then rewriting the body in a new form (possibly HttpBody.Bytes). I think we need to ensure each type of HttpBody is handled correctly.

lauzadis · 2024-11-27T19:41:28Z

.../test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptorTest.kt

@@ -126,23 +133,6 @@ class FlexibleChecksumsRequestInterceptorTest {
        assertEquals(0, call.request.headers.getNumChecksumHeaders())
    }

-    @Test
-    fun itSetsChecksumHeaderViaExecutionContext() = runTest {


Why was this test removed?

We're no longer setting the checksum header via execution context

lauzadis · 2024-11-27T19:41:52Z

...test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptorTest.kt

@@ -163,29 +168,4 @@ class FlexibleChecksumsResponseInterceptorTest {

        op.roundTrip(client, TestInput("input"))
    }
-
-    @Test
-    fun testSkipsValidationWhenDisabled() = runTest {


Why was this test removed?

This should be covered under ChecksumConfigTest > ResponseChecksumValidation but I added it back

lauzadis · 2024-11-27T19:43:48Z

...smithy-client/common/src/aws/smithy/kotlin/runtime/client/config/HttpChecksumClientConfig.kt

+
+public enum class HttpChecksumConfigOption {
+    /**
+     * SDK will create/validate checksum if the service marks it as required or if this is set.


docs: create -> calculate

lauzadis · 2024-11-27T19:54:55Z

.../test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptorTest.kt

correctness: missing tests for when requestChecksumRequired = false and requestChecksumCalculation = HttpChecksumConfigOption.WHEN_REQUIRED. Same for response validation

The request test cases are already here. I think the response validation unit tests need one more to be exhaustive.

Those tests in ChecksumConfigTest.kt aren't validating interceptor behavior like the tests in this file do.

ianbotsf · 2024-11-27T19:02:29Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

- * @param checksumAlgorithmNameInitializer an optional function which parses the input [I] to return the checksum algorithm name.
- * if not set, then the [HttpOperationContext.ChecksumAlgorithm] execution context attribute will be used.
+ * If the request will be streamed:
+ * - The checksum calculation is done asynchronously using a hashing & completing body.


Nit: "asynchronously" → "during transmission"

ianbotsf · 2024-11-27T19:03:02Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+ * - The checksum will be sent in a trailing header, once the request is consumed.
+ *
+ * If the request will not be streamed:
+ * - The checksum calculation is done synchronously


Nit: "synchronously" → "before transmission"

ianbotsf · 2024-11-27T19:31:01Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+    private val defaultChecksumAlgorithm = lazy { Crc32() }
+    private val defaultChecksumAlgorithmHeaderPostfix = "crc32"
+
+    private val checksumAlgorithm = userSelectedChecksumAlgorithm?.let {
+        val hashFunction = userSelectedChecksumAlgorithm.toHashFunction()
+        if (hashFunction == null || !hashFunction.isSupported) {
+            throw ClientException("Checksum algorithm '$userSelectedChecksumAlgorithm' is not supported for flexible checksums")
+        }
+        checksumHeader.append(userSelectedChecksumAlgorithm.lowercase())
+        hashFunction
+    } ?: if (forcedToCalculateChecksum) {
+        checksumHeader.append(defaultChecksumAlgorithmHeaderPostfix)
+        defaultChecksumAlgorithm.value
+    } else {
+        null
+    }


Nit: Seems unnecessary to declare forcedToCalculateChecksum, defaultChecksumAlgorithm, and defaultChecksumAlgorithmHeaderPostfix as fields only to use them in one if branch. Could this just be:

?: if (requestChecksumRequired || requestChecksumCalculation == HttpChecksumConfigOption.WHEN_SUPPORTED) { checksumHeader.append("crc32") Crc32() } else

ianbotsf · 2024-11-27T19:35:19Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+        userProviderChecksumHeader(context.protocolRequest, logger)?.let {
+            logger.debug { "User supplied a checksum via header, skipping checksum calculation" }


Style: These log messages which talk about the user feel awkward. Log messages are for users so it seems strange to mention the user in the third person. I'd suggest rewording these to be more about data/actions and less about the actors involved (e.g., "Found a provided checksum in the request, skipping checksum calculation").

ianbotsf · 2024-11-27T19:47:52Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

-
-                deferredChecksum.complete(checksum)
-            } else {
-                logger.debug { "Calculating checksum asynchronously" }


Nit: We've lost the log message that we're calculating a checksum ~~asynchronously~~ during transmission. I'd suggest moving your "Calculating checksum using '$checksumAlgorithm'" message into the if/else clauses, tweaking each to mention whether the checksum is being calculated before or during transmission.

ianbotsf · 2024-11-27T20:14:57Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

+private fun String.isCompositeChecksum(): Boolean {
+    // Ends with "-#" where "#" is a number between 1-1000
+    val regex = Regex("-([1-9][0-9]{0,2}|1000)$")
+    return regex.containsMatchIn(this)
+}


Nit: When this logic is made S3 specific, we shouldn't assert on the part number fitting inside a certain range. Just "-(\d)+$" should suffice.

ianbotsf · 2024-11-27T20:16:29Z

.../test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptorTest.kt

Nit: Please add tests for HttpChecksumConfigOption.WHEN_REQUIRED. Applies to response interceptors as well.

The request test cases are already here. I think the response validation unit tests need one more to be exhaustive.

ianbotsf · 2024-11-27T21:08:36Z

.../test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptorTest.kt

-    @Test
-    fun itSetsChecksumHeaderViaExecutionContext() = runTest {
-        checksums.forEach { (checksumAlgorithmName, expectedChecksumValue) ->
-            val req = HttpRequestBuilder().apply {
-                body = HttpBody.fromBytes("<Foo>bar</Foo>".encodeToByteArray())
-            }
-
-            val op = newTestOperation<Unit, Unit>(req, Unit)
-            op.context[HttpOperationContext.ChecksumAlgorithm] = checksumAlgorithmName
-            op.interceptors.add(FlexibleChecksumsRequestInterceptor<Unit>())
-
-            op.roundTrip(client, Unit)
-            val call = op.context.attributes[HttpOperationContext.HttpCallList].first()
-            assertEquals(expectedChecksumValue, call.request.headers["x-amz-checksum-$checksumAlgorithmName"])
-        }
-    }


Question: Do we still use HttpOperationContext.ChecksumAlgorithm or can it be deprecated?

I think it's still used for MD5 checksums in the httpChecksumRequiredTrait

Update on this, it's no longer used anywhere so we can deprecate/remove it

ianbotsf · 2024-11-27T21:10:10Z

...test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptorTest.kt

-    @Test
-    fun testSkipsValidationWhenDisabled() = runTest {
-        val req = HttpRequestBuilder()
-        val op = newTestOperation<TestInput>(req)
-
-        op.interceptors.add(
-            FlexibleChecksumsResponseInterceptor<TestInput> {
-                false
-            },
-        )
-
-        val responseChecksumHeaderName = "x-amz-checksum-crc32"
-
-        val responseHeaders = Headers {
-            append(responseChecksumHeaderName, "incorrect-checksum-would-throw-if-validated")
-        }
-
-        val client = getMockClient(response, responseHeaders)
-
-        val output = op.roundTrip(client, TestInput("input"))
-        output.body.readAll()
-
-        assertNull(op.context.getOrNull(ChecksumHeaderValidated))
-    }


Question: Why did we delete this test?

This should be covered under ChecksumConfigTest > ResponseChecksumValidation but I added it back

ianbotsf · 2024-11-27T21:13:01Z

...smithy-client/common/src/aws/smithy/kotlin/runtime/client/config/HttpChecksumClientConfig.kt

+    /**
+     * SDK will create/validate checksum if the service marks it as required or if this is set.
+     */
+    WHEN_SUPPORTED,


Nit: "or if this is set" → "or if the service offers optional checksums"

ianbotsf · 2024-12-04T18:08:27Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+    }
+    private val checksumAlgorithm = requestChecksumAlgorithm?.let {


Nit: Missing vertical whitespace

ianbotsf · 2024-12-09T17:15:02Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+    private val checksumAlgorithm = requestChecksumAlgorithm?.let {
+        val hashFunction = requestChecksumAlgorithm.toHashFunction()
+        if (hashFunction == null || !hashFunction.isSupported) {
+            throw ClientException("Checksum algorithm '$requestChecksumAlgorithm' is not supported for flexible checksums")
+        }
+        hashFunction
+    } ?: if (requestChecksumRequired || requestChecksumCalculation == HttpChecksumConfigOption.WHEN_SUPPORTED) {
+        Crc32()
+    } else {
+        null
+    }


Suggestion: Could be simplified a bit:

private val checksumAlgorithm = requestChecksumAlgorithm ?.let { it.toHashFunction() ?: throw ClientException("Checksum algorithm '$it' is not supported for flexible checksums") } ?.takeIf { it.isSupported } ?: (Crc32().takeIf { requestChecksumRequired || requestChecksumCalculation == HttpChecksumConfigOption.WHEN_SUPPORTED })

Or even more by extracting the throwing logic to a helper function:

// in HashFunction.kt public fun String.toHashFunctionOrThrow(): String = toHashFunction() ?: throw ClientException("Checksum algorithm '$this' is not supported for flexible checksums") // in FlexibleChecksumsRequestInterceptor.kt private val checksumAlgorithm = requestChecksumAlgorithm ?.toHashFunctionOrThrow() ?.takeIf { it.isSupported } ?: (Crc32().takeIf { requestChecksumRequired || requestChecksumCalculation == HttpChecksumConfigOption.WHEN_SUPPORTED })

ianbotsf · 2024-12-09T17:16:59Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+            request.body = request.body
+                .toHashingBody(
+                    checksumAlgorithm,
+                    request.body.contentLength,
+                )
+                .toCompletingBody(
+                    deferredChecksum,
+                )


Nit: Unnecessary wrapping

ianbotsf · 2024-12-09T17:32:38Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

-                    .toHashingBody(checksumAlgorithm, req.body.contentLength)
-                    .toCompletingBody(deferredChecksum)
-            }
+        context.executionContext.emitBusinessMetric(checksumAlgorithm.toBusinessMetric())


Question: I see here we're emitting the metric for the specific algorithm (e.g., FLEXIBLE_CHECKSUMS_REQ_CRC32C). Where are we emitting the metric for request/response enablement mode (e.g., FLEXIBLE_CHECKSUMS_REQ_WHEN_REQUIRED)?

We're emitting those metrics in the SDK side. Here and here.

ianbotsf · 2024-12-09T17:43:09Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+    private fun HttpRequest.userProvidedChecksumHeader(logger: Logger): String? {
+        this.headers.entries().forEach { header ->
+            val headerName = header.key.lowercase()
+            if (headerName.startsWith("x-amz-checksum-")) {
+                if (headerName == "x-amz-checksum-md5") {
+                    logger.debug {
+                        "MD5 checksum was supplied via header, MD5 is not a supported algorithm, ignoring header"
+                    }
+                } else {
+                    return headerName
+                }
+            }
+        }
+        return null
+    }


Suggestion: Could be simplified:

private fun HttpRequest.userProvidedChecksumHeader(logger: Logger) = headers .names() .filter { it.startsWith("x-amz-checksum-", ignoreCase = true) } .filterNot { key -> key .equals("x-amz-checksum-md5", ignoreCase = true) .also { if (it) logger.debug { "MD5 checksum was supplied via header, MD5 is not a supported algorithm, ignoring header" } } } .firstOrNull() }

ianbotsf · 2024-12-09T17:46:50Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

+
+/**
+ * Verifies if a checksum is composite.
+ * Composite checksums are used for multipart uploads.
+ */
+private fun String.isCompositeChecksum(): Boolean {
+    // Ends with "-#" where "#" is a number
+    val regex = Regex("-(\\d)+$")
+    return regex.containsMatchIn(this)
+}


Correctness: This customization should be specific to S3 only. Any other service which returns a checksum with -# in it should be handled as normal (likely by failing validation). We should move this logic/check to aws-sdk-kotlin in a customization that only applies to S3.

After reading more of the aws-sdk-kotlin PR and re-reading this interceptor, I can see now that this composite checksum usage is gated by a constructor flag. This works but it still seems too specific to a non-smithy-kotlin concern to me. I'd be interested in seeing if this logic can be completely removed from smithy-kotlin in favor of something more additive in aws-sdk-kotlin.

ianbotsf · 2024-12-09T18:18:53Z

.../test/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptorTest.kt

Those tests in ChecksumConfigTest.kt aren't validating interceptor behavior like the tests in this file do.

lauzadis · 2024-12-11T15:58:00Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

@@ -26,7 +26,7 @@ import kotlinx.coroutines.job
 import kotlin.coroutines.coroutineContext

 /**
- * Calculates a request's checksum.
+ * Handles request checksums.


clarification: "Handles request checksums for operations with the [HttpChecksumTrait] applied"

lauzadis · 2024-12-11T15:58:56Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt


-    private val checksumAlgorithm = userSelectedChecksumAlgorithm?.let {
-        val hashFunction = userSelectedChecksumAlgorithm.toHashFunction()
+    // FIXME: Remove in next minor version bump


Can you add these to our internal ticket SDK-KT-385

Sure, SDK-KT-385 seems to be more for SDK changes. I can create a ticket for the smith kotlin version bump since it seems like we'll do a minor version bump for the SDK soon and one for smithy kotlin later at a different time

I'm pretty sure we will bump both at the same time

Okay, in that case I'll just fix the FIXMEs since this should go into a v1.4 branch. I have to update the PR.

lauzadis · 2024-12-11T15:59:47Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

    private val responseValidationRequired: Boolean,
    private val responseChecksumValidation: HttpChecksumConfigOption?,
+    private val serviceUsesCompositeChecksums: Boolean,


Missing KDocs

correctness (naming): operationUsesCompositeChecksums

lauzadis · 2024-12-11T16:02:08Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

    private val responseValidationRequired: Boolean,
    private val responseChecksumValidation: HttpChecksumConfigOption?,
+    private val serviceUsesCompositeChecksums: Boolean,


question: Why do we need this extra flag to tell if the operation might send back a composite checksum? Why can't we just rely on serviceChecksumValue.isCompositeChecksum()?

Ian left a comment above, composite checksums are SDK specific so they should be handled downstream. I was using a flag to check for them but my most recent (local) changes remove this option and handle composite checksums completely in the SDK.

…flexible-checksums

lauzadis · 2024-12-13T15:45:32Z

codegen/protocol-tests/model/error-correction-tests.smithy

@@ -60,7 +64,7 @@ structure TestStruct {
    @required
    nestedListValue: NestedList

-    @required
+    // @required


Were these @required removed to fix some protocol tests? I'd rather leave them disabled than change our test models

The required trait isn't supposed to be there, a smithy validator will throw an error if it is, even if I disable the protocol test

lauzadis · 2024-12-13T15:46:55Z

...tlin/software/amazon/smithy/kotlin/codegen/rendering/protocol/HttpProtocolClientGenerator.kt

    /**
     * Render optionally installing Md5ChecksumMiddleware.
-     * The Md5 middleware will only be installed if the operation requires a checksum and the user has not opted-in to flexible checksums.
+     * The Md5 middleware will only be installed if the operation requires a checksum.
     */


question: Why is this installed for every operation now without consideration of flexible checksums? If we enable CRC32 by default, doesn't this also calculate the MD5 unnecessarily?

.../aws-signing-common/common/src/aws/smithy/kotlin/runtime/auth/awssigning/AwsSigningConfig.kt

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

lauzadis · 2024-12-13T15:50:14Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

+
+    private val checksumHeader = buildString {
+        append("x-amz-checksum-")
+        append(requestChecksumAlgorithm?.lowercase() ?: "crc32")


style: "crc32" could be lifted to a const val SDK_DEFAULT_CHECKSUM_ALGORITHM for easier understanding

lauzadis · 2024-12-13T15:53:06Z

...ommon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsRequestInterceptor.kt

-        check(context.protocolRequest.body !is HttpBody.Empty) {
-            "Can't calculate the checksum of an empty body"
+        if (checksumAlgorithm == null) {
+            logger.debug { "A checksum algorithm isn't selected and checksum calculation isn't required, skipping checksum calculation" }


We could also fall into this case when a user does select a checksum algorithm but they also configure WHEN_REQUIRED.

I think a better log would be:
"A checksum algorithm isn't selected or checksum calculation isn't required, skipping checksum calculation"

lauzadis · 2024-12-13T15:55:57Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

-        )
+        val checksumAlgorithm = checksumHeader
+            .removePrefix("x-amz-checksum-")
+            .toHashFunction() ?: throw ClientException("Could not parse checksum algorithm from header $checksumHeader")


Should use toHashFunctionOrThrow here

lauzadis · 2024-12-13T15:56:35Z

...mmon/src/aws/smithy/kotlin/runtime/http/interceptors/FlexibleChecksumsResponseInterceptor.kt

+                        .toChecksumValidatingBody(serviceChecksumValue),
+                )
+            }
+            else -> throw IllegalStateException("Http body type '$bodyType' is not supported for flexible checksums.")


nit: capitalize HTTP or make it HttpBody

lauzadis · 2024-12-13T15:57:17Z

runtime/runtime-core/common/src/aws/smithy/kotlin/runtime/Exceptions.kt

+
+/**
+ * Runtime accessible version of the [IllegalStateException]
+ */
+@InternalApi
+public class IllegalStateException(message: String) : IllegalStateException(message)


What's the purpose of this, why is not accessible from runtime already?

lauzadis · 2024-12-13T15:57:54Z

runtime/runtime-core/common/src/aws/smithy/kotlin/runtime/text/Text.kt

@@ -6,9 +6,13 @@
 package aws.smithy.kotlin.runtime.text

 import aws.smithy.kotlin.runtime.InternalApi
+import java.util.*


correctness: You can't use java imports in a common source set

…flexible-checksums

github-actions · 2024-12-19T17:44:53Z

Affected Artifacts

Changed in size

Artifact	Pull Request (bytes)	Latest Release (bytes)	Delta (bytes)	Delta (percentage)
http-jvm.jar	108,391	103,477	4,914	4.75%
smithy-client-jvm.jar	65,382	62,681	2,701	4.31%
runtime-core-jvm.jar	816,527	812,469	4,058	0.50%
aws-signing-common-jvm.jar	66,779	66,491	288	0.43%
aws-signing-default-jvm.jar	52,520	52,433	87	0.17%
http-client-jvm.jar	314,579	314,811	-232	-0.07%

lauzadis · 2024-12-19T18:39:58Z

codegen/protocol-tests/build.gradle.kts

+    // FIXME: Re-enable. This test is broken after a smithy update: https://github.com/smithy-lang/smithy/pull/2467
+    // ProtocolTest("aws-json-10", "aws.protocoltests.json10#JsonRpc10"),


FYI I know we talked about disabling some tests that are failing in Smithy 1.53.0, but these are disabling the entire test suite, which we don't want to do

lauzadis · 2024-12-19T18:43:42Z

...mithy-kotlin-codegen/src/main/kotlin/software/amazon/smithy/kotlin/codegen/model/ShapeExt.kt

@@ -32,7 +33,7 @@ import software.amazon.smithy.rulesengine.traits.EndpointTestsTrait
 * shape's closure for example)
 */
 @Suppress("EXTENSION_SHADOWED_BY_MEMBER")
-inline fun <reified T : Shape> Model.shapes(): List<T> = shapes(T::class.java).toList()
+inline fun <reified T : Shape> Model.shapes(): List<T> = shapes(T::class.java).kotlinToList()


This has never been a problem before, what changed?

There was a refactor in AWS SDK Kotlin to make the codegen tests KMP, which led to the JVM CI checks failing because we've been using JVM 17. I set the JVM version in the tests to 1.8 and the toList function used above is only available since JVM 16

Ok that makes sense. You should be able to use shapes(T::class.java).collect(Collectors.toList()) instead of hacking around with the Kotlin toList

lauzadis · 2024-12-19T18:44:10Z

...software/amazon/smithy/kotlin/codegen/rendering/checksums/HttpChecksumRequiredIntegration.kt

+import software.amazon.smithy.model.traits.HttpChecksumRequiredTrait
+
+/**
+ * Handles the `httpChecksumRequired` trait.


nit/docs: Explain more about how it "handles" the trait

The "meat" of the integration is the middleware and both of them have their own Kdocs. It feels like docs overkill to also add a summary of what each middleware is doing here

lauzadis · 2024-12-19T18:45:37Z

...software/amazon/smithy/kotlin/codegen/rendering/checksums/HttpChecksumRequiredIntegration.kt

+        writer.write(
+            "op.context[#T.DefaultChecksumAlgorithm] = #S",
+            RuntimeTypes.HttpClient.Operation.HttpOperationContext,
+            "MD5",


question/correctness: Can/should we store this as a HashFunction rather than a String?

We could, I think we would have to initialize the HashFunction in the context and its seems slightly worse than what we have right now. Thoughts?

lauzadis · 2024-12-19T18:48:49Z

...client/common/src/aws/smithy/kotlin/runtime/http/interceptors/AbstractChecksumInterceptor.kt

+
+/**
+ * @return The default checksum algorithm name, null if default checksums are disabled.
+ */
+internal fun defaultChecksumAlgorithmName(context: ProtocolRequestInterceptorContext<Any, HttpRequest>): String? =
+    context.executionContext.getOrNull(HttpOperationContext.DefaultChecksumAlgorithm)


correctness: This seems like the wrong place to define this function

Where would be better?

I would have said HttpChecksumRequiredInterceptor but I see it's also used by FlexibleChecksumsRequestInterceptor. I think it's fine here.

style: make it an extension val

internal val ProtocolRequestInterceptorContext<Any, HttpRequest>.defaultChecksumAlgorithmName: String? get() = executionContext.getOrNull(HttpOperationContext.DefaultChecksumAlgorithm)

lauzadis · 2024-12-19T19:12:13Z

runtime/protocol/http/common/src/aws/smithy/kotlin/runtime/http/HttpBody.kt

+/**
+ * Convert an [HttpBody] with an underlying [HashingSource] or [HashingByteReadChannel]
+ * to a [CompletingSource] or [CompletingByteReadChannel], respectively.
+ */
+@InternalApi
+public fun HttpBody.toCompletingBody(deferred: CompletableDeferred<String>): HttpBody = when (this) {


These used to be private, now they are @InternalApi public, meaning they are included in our API dumps and part of backwards compatibility. Was this move necessary?

Oops, I moved this here because I was going to add support for streaming checksums in the HttpCheksumsRequired interceptor

lauzadis · 2024-12-19T19:12:59Z

runtime/runtime-core/common/src/aws/smithy/kotlin/runtime/hashing/HashFunction.kt

+ */
+@InternalApi
+public val HashFunction.isSupportedForFlexibleChecksums: Boolean get() =
+    algorithmsSupportedForFlexibleChecksums.contains(this::class.simpleName)


correctness: Relying on this::class.simpleName seems unsafe.

Better solution:

@InternalApi public val HashFunction.isSupportedForFlexibleChecksums: Boolean get() = when (this) { is Crc32, is Crc32c, is Sha1, is Sha256 -> true else -> false }

Yeah agree, that's what we had before but we would have to keep track of what checksum algorithms are supported for flexible checksums in two places. In HashFunction.isSupportedForFlexibleChecksums & algorithmsSupportedForFlexibleChecksums.

I made a tradeoff here, it seems more risky to possibly miss updating what algorithms we support in both locations than relying on the class name. Plus we have unit tests that should lessen some of our concerns here

The only other use of algorithmsSupportedForFlexibleChecksums is in a log message. I'd rather remove the log message than keep this class reflection

lauzadis · 2024-12-19T19:15:03Z

runtime/runtime-core/common/src/aws/smithy/kotlin/runtime/util/Concurrency.kt

+@InternalApi
+public fun runBlocking(block: suspend () -> Unit) {
+    runBlocking {
+        block()
+    }
+}


question: What is this used for?

It's so we can have runBlocking as a runtime type, it's used in the presigner generator

You can just add a new runBlocking type under the existing KotlinxCoroutines runtime type

lauzadis · 2024-12-19T19:15:27Z

...smithy-client/common/src/aws/smithy/kotlin/runtime/client/config/HttpChecksumClientConfig.kt

@@ -0,0 +1,40 @@
+package aws.smithy.kotlin.runtime.client.config


correctness: missing license at the top. same applies to Concurrency.kt

Thanks, I think we need to adjust our Ktlint for be more stringent when it comes to license headers

lauzadis · 2024-12-19T19:16:11Z

...smithy-client/common/src/aws/smithy/kotlin/runtime/client/config/HttpChecksumClientConfig.kt

+    }
+}
+
+public enum class HttpChecksumConfigOption {


docs: missing KDocs

Maybe we can add a new rule to Ktlint about Kdocs? Not sure it that's possible but I think I'd be helpful

lauzadis · 2024-12-19T19:30:47Z

...smithy-client/common/src/aws/smithy/kotlin/runtime/client/config/HttpChecksumClientConfig.kt

+    }
+}
+
+public enum class HttpChecksumConfigOption {


correctness: Right now request and response checksum configs both use this enum. In the future, their config options might diverge, which will force us to make an API break. I'd recommend splitting this into HttpChecksumRequestConfigOption and HttpChecksumResponseConfigOption now to get ahead of that.

It seems like just speculation for now, I say we only separate them if that time comes. Changing the type of this config option would be a breaking change for all SDKs. I think it's more likely that we would make additive changes

It's not speculative, there was some discussion during the spec review about adding more options to either request or response but not both.

The spec calls out RequestChecksumCalculation and ResponseChecksumValidation as separate config types, so adding a new config option to either of those wouldn't be a breaking change for SDKs which implement it as the spec says.

0marperez added 10 commits November 10, 2024 22:28

Bump smithy IDL version

2a9d104

Signed-off-by: 0marperez <[email protected]>

Add requestChecksumCalculation config option

205839c

Added responseChecksumValidation

7233b71

Add todos for business metrics

e1dc616

Unit tests pass

e436482

Merge branch 'main' of https://github.com/awslabs/smithy-kotlin into …

abdba02

…flexible-checksums

E2E tests pass

3e4c891

Self review

9760ee1

Self review 2

f8b39b0

Smithy codegen version bump

f676b7b

0marperez added the no-changelog Indicates that a changelog entry isn't required for a pull request. Use sparingly. label Nov 27, 2024

This comment has been minimized.

Sign in to view

lauzadis reviewed Nov 27, 2024

View reviewed changes

ianbotsf reviewed Nov 27, 2024

View reviewed changes

0marperez and others added 4 commits December 3, 2024 11:23

Make composite checksum check S3 specific

6e9b206

Turn off all failing protocol tests

fb3a52a

PR feedback and fix breaking changes

40bb298

Merge branch 'main' into flexible-checksums

828adaa

This comment has been minimized.

Sign in to view

Trigger CI

e2068e7

This comment has been minimized.

Sign in to view

Drop support for http body dot bytes response checksums

08b4a37

This comment has been minimized.

Sign in to view

0marperez marked this pull request as ready for review December 4, 2024 20:53

0marperez requested a review from a team as a code owner December 4, 2024 20:53

Fix HttpChecksumRequiredTrait

91355d1

This comment has been minimized.

Sign in to view

Fix kotlin writer runtime exception

1fcd4b2

This comment has been minimized.

Sign in to view

Deprecate HttpOperationContext.ChecksumAlgorithm

4edabfb

ianbotsf requested changes Dec 9, 2024

View reviewed changes

lauzadis reviewed Dec 11, 2024

View reviewed changes

PR feedback

db65d74

This comment has been minimized.

Sign in to view

0marperez added 3 commits December 12, 2024 22:07

Re-add support for http body dot bytes response checksusms

b2e6f61

Presigned URL checksums

c63cf83

Merge branch 'main' of https://github.com/awslabs/smithy-kotlin into …

b68a9f5

…flexible-checksums

lauzadis reviewed Dec 13, 2024

View reviewed changes

0marperez added 4 commits December 13, 2024 15:02

PR feedback checkpoint?

0dd7886

Refactor checksum interceptors

d74c949

Fix composite checksums

1157f26

Make it compile

d9f0659

This comment has been minimized.

Sign in to view

Merge branch 'main' of https://github.com/awslabs/smithy-kotlin into …

dc3ce8b

…flexible-checksums

This comment has been minimized.

Sign in to view

Use toList supported for JVM versions less than 16

3ef1c20

lauzadis requested changes Dec 19, 2024

View reviewed changes

lauzadis reviewed Dec 19, 2024

View reviewed changes

		userProviderChecksumHeader(context.protocolRequest, logger)?.let {
		logger.debug { "User supplied a checksum via header, skipping checksum calculation" }

		}
		private val checksumAlgorithm = requestChecksumAlgorithm?.let {

		// FIXME: Re-enable. This test is broken after a smithy update: https://github.com/smithy-lang/smithy/pull/2467
		// ProtocolTest("aws-json-10", "aws.protocoltests.json10#JsonRpc10"),

		@@ -0,0 +1,40 @@
		package aws.smithy.kotlin.runtime.client.config

feat: support default checksums #1191

Are you sure you want to change the base?

feat: support default checksums #1191

Conversation

0marperez commented Nov 27, 2024

Issue #

Description of changes

This comment has been minimized.

This comment has been minimized.

0marperez commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lauzadis Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0marperez Dec 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

Choose a reason for hiding this comment

0marperez Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

lauzadis Dec 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment has been minimized.

lauzadis Nov 27, 2024 •

edited

Loading

0marperez Dec 2, 2024 •

edited

Loading

0marperez Dec 13, 2024 •

edited

Loading

lauzadis Dec 13, 2024 •

edited

Loading