Move sdk.NowTime out of Internal Package or Make Configurable

[austinarbor]

Describe the feature

If the system running the aws sdk code has a clock that is not in sync with real time, it will suffer from clock skew issues, namely with request signature signing and credential expiration. This will cause the requests to fail with 403 errors. There isn't a clear solution in the GO SDK to handle this scenario automatically that I'm aware of. I did see some code in the retry logic to calculate skew, but it wasn't apparent to me if the skew was taken into account on the retried request. Even if the skew is used in the retry, it would be more efficient to use the skew before the request is made to reduce the number of retries required.

By moving sdk.NowTime out of the internal package or making it configurable via options, this allows us to provide our own clock skew fixes for all code paths without needing to provide separate fixes for credential expiration, signature signing, etc.

Use Case

Consider a system that has a clock 15 minutes behind real time. When making requests:

• v4 signature will fail because time.Now is too old
• when using credentials that expire, the credentials will not refresh in time unless the expiry window is large enough to cover the clock skew of the system

In order to fix both issues, we must provide:

• Custom HTTPSignerV4 implementation to correct clock skew. This must be passed into the client for every service we are using
• Override default retry behavior to check for ExpiredToken error code, then invalidate the credentials before the next retry

Proposed Solution

Move sdk.NowTime out of the internal package, or make it configurable via options. This will allow users of the sdk to provide a single fix for all potential clock skew errors

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS Go SDK V2 Module Versions Used

github.com/aws/[email protected] github.com/aws/[email protected]
github.com/aws/[email protected] github.com/google/[email protected]
github.com/aws/[email protected] github.com/jmespath/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/aws-sdk-go-v2/service/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/feature/ec2/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/internal/[email protected] github.com/google/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/internal/endpoints/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/aws-sdk-go-v2/service/internal/[email protected]
github.com/aws/aws-sdk-go-v2/service/[email protected] github.com/aws/[email protected]
github.com/aws/[email protected] github.com/aws/[email protected]
github.com/aws/[email protected] github.com/google/[email protected]
github.com/aws/[email protected] github.com/jmespath/[email protected]

Go version used

go version go1.19.3 darwin/arm64

[lucix-aws]

I've spawned #2321 to add automatic skew detection.

I don't view this as justification to expose wall-clock time config in the SDK right now, skew detection solves this problem automatically at a cost of 1 round trip per service client in the worst case scenario (the skew is recognized once and will persist on all future requests made in that client).

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.