SecretManager | List secrets on ap-northeast-1: connect ETIMEDOUT/ENETUNREACH

[loganmzz]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

I have written a tool trying to resolve existing Terraform resource.

Everything is working fine with many services (Secret Manager, Security Group, EventBridge) and two regions (eu-central-1 and ap-northeast-1) but listing secrets on ap-northeast-1.

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

It results in:

AggregateError
        at internalConnectMultiple (node:net:1114:18)
        at internalConnectMultiple (node:net:1177:5)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      code: 'ETIMEDOUT',
      name: 'TimeoutError',
      '$metadata': { attempts: 3, totalRetryDelay: 209 },
      [errors]: [
        Error: connect ETIMEDOUT 18.179.209.155:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.209.155',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
          port: 443
        },
        Error: connect ETIMEDOUT 18.179.211.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '18.179.211.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
          port: 443
        },
        Error: connect ETIMEDOUT 52.198.52.244:443
            at createConnectionError (node:net:1634:14)
            at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -110,
          code: 'ETIMEDOUT',
          syscall: 'connect',
          address: '52.198.52.244',
          port: 443
        },
        Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
            at internalConnectMultiple (node:net:1176:40)
            at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
            at listOnTimeout (node:internal/timers:575:11)
            at process.processTimers (node:internal/timers:514:7) {
          errno: -101,
          code: 'ENETUNREACH',
          syscall: 'connect',
          address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
          port: 443
        }
      ]
    }

Tried with AWS CLI I have no issue:

AWS_REGION=ap-northeast-1 aws secretsmanager list-secrets --filters 'Key=name,Values=...'

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/[email protected]

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

v20.10.0

Reproduction Steps

const sm = new SecretsManager({region: 'ap-northeast-1'});
const listSecrets = await sm.listSecrets();

Observed Behavior

Error:

AggregateError
    at internalConnectMultiple (node:net:1114:18)
    at internalConnectMultiple (node:net:1177:5)
    at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
    at listOnTimeout (node:internal/timers:575:11)
    at process.processTimers (node:internal/timers:514:7) {
  code: 'ETIMEDOUT',
  name: 'TimeoutError',
  '$metadata': { attempts: 3, totalRetryDelay: 157 },
  [errors]: [
    Error: connect ETIMEDOUT 18.179.209.155:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.209.155',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a00:e3de:c82b:7b57:b9a4:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a00:e3de:c82b:7b57:b9a4',
      port: 443
    },
    Error: connect ETIMEDOUT 18.179.211.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '18.179.211.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a02:4b65:ecb5:dfe8:2f13',
      port: 443
    },
    Error: connect ETIMEDOUT 52.198.52.244:443
        at createConnectionError (node:net:1634:14)
        at Timeout.internalConnectMultipleTimeout (node:net:1685:38)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -110,
      code: 'ETIMEDOUT',
      syscall: 'connect',
      address: '52.198.52.244',
      port: 443
    },
    Error: connect ENETUNREACH 2406:da14:afa:8a01:bfa8:ff5e:a40a:74af:443 - Local (:::0)
        at internalConnectMultiple (node:net:1176:40)
        at Timeout.internalConnectMultipleTimeout (node:net:1687:3)
        at listOnTimeout (node:internal/timers:575:11)
        at process.processTimers (node:internal/timers:514:7) {
      errno: -101,
      code: 'ENETUNREACH',
      syscall: 'connect',
      address: '2406:da14:afa:8a01:bfa8:ff5e:a40a:74af',
      port: 443
    }
  ]
}

Expected Behavior

Valid HTTP response

Possible Solution

N/A

Additional Information/Context

No response

[zshzbh]

Hey @loganmzz ,

Thanks for the feedback! However I can't reproduce this issue.

This is the code I have -

import { SecretsManager } from '@aws-sdk/client-secrets-manager';

const client = new SecretsManager({region: 'ap-northeast-1'});

const response = await client.listSecrets();
console.log(response)

And the result is 200.

{
  '$metadata': {
    httpStatusCode: 200,
    requestId: 'f40c7c36-63d2-459b-a6e3-XXX',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  },
  SecretList: [
    {
      ARN: 'arn:aws:secretsmanager:ap-northeast-1:XXX:secret:prod/test/secret-XXX',
      CreatedDate: 2024-12-04T23:07:09.893Z,
      LastChangedDate: 2024-12-04T23:07:10.145Z,
      Name: 'prod/test/secret',
      SecretVersionsToStages: [Object],
      Tags: []
    }
  ]
}

This error indicates a network timeout when trying to connect to AWS Secrets Manager. You can manually setup timeout and retry attempts -

const client = new SecretsManager({
    region: 'ap-northeast-1',
    logger: console,
    maxAttempts: 3,
    retryMode: 'standard',
    requestTimeout: 3000,
    connectTimeout: 3000
});

Or if you think I miss anything, please add additional info.

Thanks!

[loganmzz]

Looks like timeouts are ignored. I increased them to 50000 and I still facing same issue in only 3s.

I run it several times (~ 10), and it passed once... May be just need to "really" increase timeout :( Any other way to increase them?

[loganmzz]

Just to be sure the information is shared, I have no issue with CLI (Python?)

[loganmzz]

@zshzbh At stated previously your setup doesn't seem to affect timeout. Any other suggestion on how to setup custom timeouts?

[zshzbh]

There's some steps I recommend -

1. Checking secrets in ap-northeast-1:
   I don't actually have access to check AWS secrets or configurations. You'll need to verify this in your own AWS account using the AWS CLI or console.

2. Terraform and network connectivity:
   You're right that Terraform is a third-party tool. When using it with AWS, it's a good practice to check network connectivity. You can do this by:

   • Ensuring your local machine can reach AWS endpoints
   • Verifying your AWS credentials are correctly configured
   • Running a simple AWS CLI command to test connectivity

3. Adding middleware to the code:
   The middleware you provided looks good. It will log the context, input, and output of AWS SDK calls. Here's how you can add it to your AWS SDK client:

import { S3Client } from "@aws-sdk/client-s3";

const client = new S3Client({ region: "ap-northeast-1" });

client.middlewareStack.add(
  (next, context) => async (args) => {
    console.log("AWS SDK context", context.clientName, context.commandName);
    console.log("AWS SDK request input", args.input);
    const result = await next(args);
    console.log("AWS SDK request output:", result.output);
    return result;
  },
  {
    name: "MyMiddleware",
    step: "build",
    override: true,
  }
);

4. Setting up higher max attempts:
   You can increase the maximum number of retry attempts by configuring the retry strategy. Here's an example:

import { S3Client } from "@aws-sdk/client-s3";
import { NodeHttpHandler } from "@aws-sdk/node-http-handler";
import { RetryStrategy } from "@aws-sdk/util-retry";

const client = new S3Client({
  region: "ap-northeast-1",
  maxAttempts: 5,  // Increase this number as needed
  requestHandler: new NodeHttpHandler({
    connectionTimeout: 5000,  // 5 seconds
    socketTimeout: 5000,  // 5 seconds
  }),
  retryStrategy: new RetryStrategy(() => Promise.resolve(3000)),  // 3 seconds delay between retries
});

5. Sharing reproducible code:
   If the above steps don't resolve the issue, please provide:
   • The complete Terraform configuration file
   • Any relevant AWS SDK code you're using
   • Full error messages and stack traces
   • The exact CLI commands you're running
   • Any environment variables or AWS credentials setup