Add retry strategy options to RemoteProviderConfig

[dmattia]

Is your feature request related to a problem? Please describe.

Occasionally, the AWS metadata services will fail to fetch credentials. One common cause I've seen is the status code 429 TooManyRequestsException, which means too many requests are sent in a short time period and the client is rate limited.

The RemoteProviderConfig type allows passing a maxRetries value to each client, however, it does not give options for how those retries should occur.

Here's an image of a trace where I had a client get rate limited with maxRetries set to 5. Notice, all of the calls fail, as all of the calls are separated by under 1ms. If the metadata endpoint is failing because of rate limiting issues, it is unlikely to not be rate limited .5ms later.

Describe the solution you'd like

I want to have an additional field to specify a retry strategy. This could be as simple as a retryDelay parameter that takes in how much time to wait for linear retries, or could be a set of fields that specifies if the retry should be exponential/linear, etc.

Describe alternatives you've considered

none

Additional context

none

[trivikr]

@dmattia We have an existing option to set retry mode for clients. It's not documented in developer guide, so I'll create a request with writers. It is similar to AWS CLI retries

We will evaluate if this retry mode can be used for while fetching credentials.

In the mean time, I created two other reports based on your feature request:

• Cache IMDS metadata token till it expires #2707
• Replace unused maxRetries with maxAttempts #2709

Do you have information on the path of http request which is throwing 429 TooManyRequestsException?

[dmattia]

Thanks @trivikr! We sure do, the full url from our trace is http://169.254.170.2/v2/credentials/<uuid_v4>.

[trivikr]

The full url from our trace is http://169.254.170.2/v2/credentials<uuid_v4>.

Oh, you're using CMDS in a Fargate container and not IMDS.
Code for future reference:

aws-sdk-js-v3/packages/credential-provider-imds/src/fromContainerMetadata.ts

Lines 58 to 63 in 4f02b78

	if (process.env[ENV_CMDS_RELATIVE_URI]) {
	return {
	hostname: CMDS_IP,
	path: process.env[ENV_CMDS_RELATIVE_URI],
	};
	}

I don't have context on CMDS, but using retry mode in those requests would help.
I'll revisit this issue when I get opportunity to deep dive into fromContainerMetadata implementation.

[trivikr]

Do update this issue if you write your own custom fromContainerMetadata implementation with custom retries.

Existing implementation:

aws-sdk-js-v3/packages/credential-provider-imds/src/fromContainerMetadata.ts

Lines 19 to 30 in 4f02b78

	export const fromContainerMetadata = (init: RemoteProviderInit = {}): CredentialProvider => {
	const { timeout, maxRetries } = providerConfigFromInit(init);
	return () =>
	retry(async () => {
	const requestOptions = await getCmdsUri();
	const credsResponse = JSON.parse(await requestFromEcsImds(timeout, requestOptions));
	if (!isImdsCredentials(credsResponse)) {
	throw new CredentialsProviderError("Invalid response received from instance metadata service.");
	}
	return fromImdsCredentials(credsResponse);
	}, maxRetries);
	};

You can write your own implementation and pass it while creating client:

import { FooClient } from "@aws-sdk/client-foo";

const client = new FooClient({
  credentials: fromContainerMetadataCustom({
    // custom params if required.
  })
});

[trivikr]

@dmattia Are you making lot of concurrent calls, and facing this issue in v3.27.0?

If yes, we're going to push a fix in v3.28.0 on Friday 08/27
The fix can be viewed in PR #2716

If you're using v3.27.0, you can temporarily downgrade to v3.26.0 while fix is published.

[aldegoeij]

related to my issue on the SecurityHub client: #2741

[github-actions]

Greetings! We’re closing this issue because it has been open a long time and hasn’t been updated in a while and may not be getting the attention it deserves. We encourage you to check if this is still an issue in the latest release and if you find that this is still a problem, please feel free to comment or open a new issue.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.