add doc support for S3 object integrity feature #2040
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aajtodd
requested changes
Feb 27, 2023
|
|
||
| If `ChecksumAlgorithm` field, and an algorithm `Checksum*` member is not set, then the SDK will compute the corresponding checksum for the request and will send it in the appropriate `x-amz-checksum-*` header in the REST request. The only distinction is if the checksum is sent as a header or trailer based on the endpoint scheme. | ||
|
|
||
| * If the scheme is HTTPS the computed checksum will be sent in a trailer header block using aws-chunked encoding. It will also do so use an unsigned payload variant called `STREAMING-UNSIGNED-PAYLOAD-TRAILER` which does not require signing each aws-chunk. The Go SDK only supports this variant of aws-chunked, which is why computed checksums sent via a aws-chunked trailer requires an HTTPS endpoint in this SDK. For trailer streaming the content-length of the `PutObject` body must be known. So either the `PutObjectInput` specifies `ContentLength` field, the `Body` field implements `interface{ Len() int }`, or the body is seekable so SDK can figure out the length to be sent. The SDK may add support for the signed aws-chunked variants (e.g. `STREAMING-AWS4-HMAC-SHA256-PAYLOAD-TRAILER`) in the future as to support sending of signed checksum trailers to non-HTTPS endpoints. That feature is supported in SDKs that had already supported signed aws-chunked payloads previously. For more information about checksum location and trailing checksum support for unsigned payload, please refer to [flexible-checksums](https://code.amazon.com/packages/AwsDrSeps/blobs/main/--/seps/accepted/shared/flexible-checksums.md) |
There was a problem hiding this comment.
this isn't true, where are you getting this information?
There was a problem hiding this comment.
From the issue related comment https://github.com/orgs/aws/projects/100/views/2?pane=issue&itemId=20470614
There was a problem hiding this comment.
Ok we can talk about that, I'm not 100% sure that is all accurate but it's also talking about S3 integrity features for PutObject/UploadPart operations. We can document those in a different section and link to them from here. This section is on the S3 transfer manager which I think we should keep the discussion here related to the differences.
| @@ -101,6 +101,48 @@ before sending it to {{% alias service="S3" %}}. `Uploader` calculates the expec | |||
| file to {{% alias service="S3" %}}. If the current value of `PartSize` requires more than 10,000 parts to upload the | |||
| file, `Uploader` increases the part size value so that fewer parts are required. | |||
|
|
|||
| ### S3 Object Integrity Features (checksum and md5) | |||
There was a problem hiding this comment.
I think this is overall focused a bit too much on the feature itself, instead I think we want to just document the differences/behavior w.r.t ContentMD5/Checksum*/ChecksumAlgorithm settings. Perhaps we should work this out in a quip doc first rather than going back and forth in PR but here is a first stab at it:
S3 [Object Integrity Checks](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html) in {{% alias service="S3" %}} can be used with the transfer manager `Upload` function.
\```go
result, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
Bucket: aws.String("my-bucket"),
Key: aws.String("my-object-key"),
Body: uploadFile,
ChecksumAlgorithm: types.ChecksumAlgorithmSha256,
})
\```
There are some differences from `PutObject` and `UploadPart` to be aware of. It is not recommended to set explicit `ContentMD5` or any of the `Checksum*` values (e.g. `ChecksumCRC32`, `ChecksumCRC32C`, `ChecksumSHA1`, or `ChecksumSHA256`) directly when using the `Upload` function. This is because the `Uploader` will internally convert large upload requests to a multipart upload for performance. The provided checksum values would be invalid for individual parts and thus are ignored by the SDK. The recommended way to enable object integrity checks is to set a `ChecksumAlgorithm` which will be used for single and multi-part uploads.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add dev guide doc support for S3 object integrity feature to resolve #1689, including checksum/md5 header configuration under SDK and http/https PutObject request.