Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules before 11/15/23 are incompatible against newer root modules #2370

Closed
khorii-cover opened this issue Nov 16, 2023 · 33 comments
Labels
compatibility Issues with SDK, Go language version, or platform compatibility.

Comments

@khorii-cover
Copy link

khorii-cover commented Nov 16, 2023

This discussion has been locked to preserve visibility of the solution, see the final comment accordingly. Original issue description follows...

Describe the bug

I am experiencing a not found, ResolveEndpointV2 error.
This error occurs during the process of marshaling a JSON object and passing it to the SQS SendMessage method.

b, err := json.Marshal(*data)
if err != nil {
  return err
}

input := &sqs.SendMessageInput{
    MessageBody: aws.String(string(b)),
    QueueUrl:    aws.String(url),
}

if _, err := n.client.SendMessage(ctx, input); err != nil {
    return err
}

Expected Behavior

The SendMessage method is expected to execute without errors.

Current Behavior

When executing the SendMessage method, the following error occurs:

not found, ResolveEndpointV2

Reproduction Steps

  1. Create a JSON payload similar to the one shown below.
  2. Use the AWS SDK for Go to create an SQS client.
  3. Use the SendMessage method of the SQS client to send the JSON payload to a specified queue.
  4. Observe the ResolveEndpointV2 error during the execution of the SendMessage call.
{
   "target_id": "example-id",
   "target_body": "Example content",
   "target_type": 0,
   "event_type": 0,
   "deep_link_url": "exampleapp:///threads/example-id",
   "image_url": "https://example.com/icon.png",
   "tokens": null
}

Possible Solution

No response

Additional Information/Context

The issue does not occur with previous versions of the SDK, and reverting to an earlier version resolves the issue.

github.com/aws/aws-sdk-go v1.47.11
github.com/aws/aws-sdk-go-v2 v1.22.2
github.com/aws/aws-sdk-go-v2/config v1.25.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.42.1
github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89

AWS Go SDK V2 Module Versions Used

module cpf-api-service

go 1.21

require (
	github.com/aws/aws-sdk-go v1.47.12
	github.com/aws/aws-sdk-go-v2 v1.23.0
	github.com/aws/aws-sdk-go-v2/config v1.25.1
	github.com/aws/aws-sdk-go-v2/service/s3 v1.42.2
	github.com/awsdocs/aws-doc-sdk-examples/gov2/s3 v0.0.0-20230216203034-1f666e8ece89
	github.com/coreos/go-oidc/v3 v3.7.0
	github.com/danielgtaylor/huma v1.14.2
	github.com/gavv/httpexpect/v2 v2.16.0
	github.com/go-redis/redis/v8 v8.11.5
	github.com/go-sql-driver/mysql v1.7.1
	github.com/google/uuid v1.4.0
	github.com/google/wire v0.5.0
	github.com/kelseyhightower/envconfig v1.4.0
	github.com/lestrrat-go/jwx v1.2.26
	github.com/mitchellh/mapstructure v1.5.0
	github.com/pkg/errors v0.9.1
	github.com/uptrace/bun v1.1.16
	github.com/uptrace/bun/dbfixture v1.1.16
	github.com/uptrace/bun/dialect/mysqldialect v1.1.16
	github.com/uptrace/bun/extra/bundebug v1.1.16
	github.com/urfave/cli/v2 v2.25.7
	go.uber.org/zap v1.26.0
	goa.design/goa/v3 v3.14.0
	goa.design/plugins/v3 v3.14.0
	golang.org/x/oauth2 v0.14.0
	google.golang.org/api v0.150.0
	gopkg.in/DataDog/dd-trace-go.v1 v1.57.0
)

require (
	cloud.google.com/go v0.110.8 // indirect
	cloud.google.com/go/compute v1.23.1 // indirect
	cloud.google.com/go/firestore v1.13.0 // indirect
	cloud.google.com/go/iam v1.1.3 // indirect
	cloud.google.com/go/storage v1.30.1 // indirect
	firebase.google.com/go/v4 v4.12.1
	github.com/MicahParks/keyfunc v1.9.0 // indirect
	golang.org/x/text v0.14.0
	gopkg.in/yaml.v2 v2.4.0
)

require (
	github.com/avast/retry-go v3.0.0+incompatible
	github.com/aws/smithy-go v1.17.0
	github.com/google/go-cmp v0.6.0
	github.com/hololive/cpf-push-notification-service/go v0.0.0-20230306103827-2389f7bbb115
	github.com/nicksnyder/go-i18n/v2 v2.2.2
	github.com/rivo/uniseg v0.4.4
	github.com/spf13/cobra v1.8.0
	github.com/stretchr/testify v1.8.4
	golang.org/x/exp v0.0.0-20230321023759-10a507213a29
)

require (
	cloud.google.com/go/compute/metadata v0.2.3 // indirect
	cloud.google.com/go/longrunning v0.5.2 // indirect
	github.com/AnatolyRugalev/goregen v0.1.0 // indirect
	github.com/DataDog/appsec-internal-go v1.0.0 // indirect
	github.com/DataDog/datadog-agent/pkg/obfuscate v0.48.0 // indirect
	github.com/DataDog/datadog-agent/pkg/remoteconfig/state v0.48.1 // indirect
	github.com/DataDog/datadog-go/v5 v5.3.0 // indirect
	github.com/DataDog/go-libddwaf v1.5.0 // indirect
	github.com/DataDog/go-tuf v1.0.2-0.5.2 // indirect
	github.com/DataDog/gostackparse v0.7.0 // indirect
	github.com/DataDog/sketches-go v1.4.2 // indirect
	github.com/Microsoft/go-winio v0.6.1 // indirect
	github.com/TylerBrock/colorjson v0.0.0-20200706003622-8a50f05110d2 // indirect
	github.com/ajg/form v1.5.1 // indirect
	github.com/andybalholm/brotli v1.0.6 // indirect
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.5.1 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.16.1 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.14.4 // indirect
	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.11.38 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.5.3 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.7.0 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.10.1 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.2.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.10.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.16.3 // indirect
	github.com/aws/aws-sdk-go-v2/service/sqs v1.24.4 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.17.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.19.2 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.25.2 // indirect
	github.com/cespare/xxhash/v2 v2.2.0 // indirect
	github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
	github.com/decred/dcrd/dcrec/secp256k1/v4 v4.2.0 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/dimfeld/httppath v0.0.0-20170720192232-ee938bf73598 // indirect
	github.com/dustin/go-humanize v1.0.1 // indirect
	github.com/ebitengine/purego v0.5.0-alpha.1 // indirect
	github.com/fatih/color v1.15.0 // indirect
	github.com/fatih/structs v1.1.0 // indirect
	github.com/go-chi/chi/v5 v5.0.10 // indirect
	github.com/go-jose/go-jose/v3 v3.0.0 // indirect
	github.com/gobwas/glob v0.2.3 // indirect
	github.com/goccy/go-json v0.10.2 // indirect
	github.com/golang-jwt/jwt/v4 v4.5.0 // indirect
	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
	github.com/golang/protobuf v1.5.3 // indirect
	github.com/google/go-querystring v1.1.0 // indirect
	github.com/google/pprof v0.0.0-20230817174616-7a8ec2ada47b // indirect
	github.com/google/s2a-go v0.1.7 // indirect
	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
	github.com/googleapis/gax-go/v2 v2.12.0 // indirect
	github.com/gorilla/websocket v1.5.0 // indirect
	github.com/imkira/go-interpol v1.1.0 // indirect
	github.com/inconshreveable/mousetrap v1.1.0 // indirect
	github.com/jinzhu/inflection v1.0.0 // indirect
	github.com/jmespath/go-jmespath v0.4.0 // indirect
	github.com/klauspost/compress v1.17.1 // indirect
	github.com/lestrrat-go/backoff/v2 v2.0.8 // indirect
	github.com/lestrrat-go/blackmagic v1.0.1 // indirect
	github.com/lestrrat-go/httpcc v1.0.1 // indirect
	github.com/lestrrat-go/iter v1.0.2 // indirect
	github.com/lestrrat-go/option v1.0.1 // indirect
	github.com/manveru/faker v0.0.0-20171103152722-9fbc68a78c4d // indirect
	github.com/mattn/go-colorable v0.1.13 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/mitchellh/go-wordwrap v1.0.1 // indirect
	github.com/outcaste-io/ristretto v0.2.3 // indirect
	github.com/philhofer/fwd v1.1.2 // indirect
	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
	github.com/richardartoul/molecule v1.0.1-0.20221107223329-32cfee06a052 // indirect
	github.com/russross/blackfriday/v2 v2.1.0 // indirect
	github.com/sanity-io/litter v1.5.5 // indirect
	github.com/secure-systems-lab/go-securesystemslib v0.7.0 // indirect
	github.com/sergi/go-diff v1.3.1 // indirect
	github.com/spaolacci/murmur3 v1.1.0 // indirect
	github.com/spf13/pflag v1.0.5 // indirect
	github.com/tinylib/msgp v1.1.8 // indirect
	github.com/tmthrgd/go-hex v0.0.0-20190904060850-447a3041c3bc // indirect
	github.com/valyala/bytebufferpool v1.0.0 // indirect
	github.com/valyala/fasthttp v1.50.0 // indirect
	github.com/vmihailenco/msgpack/v5 v5.3.5 // indirect
	github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
	github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
	github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
	github.com/xeipuuv/gojsonschema v1.2.0 // indirect
	github.com/xrash/smetrics v0.0.0-20201216005158-039620a65673 // indirect
	github.com/yalp/jsonpath v0.0.0-20180802001716-5cc68e5049a0 // indirect
	github.com/yudai/gojsondiff v1.0.0 // indirect
	github.com/yudai/golcs v0.0.0-20170316035057-ecda9a501e82 // indirect
	go.opencensus.io v0.24.0 // indirect
	go.uber.org/atomic v1.11.0 // indirect
	go.uber.org/multierr v1.10.0 // indirect
	go4.org/intern v0.0.0-20230525184215-6c62f75575cb // indirect
	go4.org/unsafe/assume-no-moving-gc v0.0.0-20230525183740-e7c30c78aeb2 // indirect
	golang.org/x/crypto v0.15.0 // indirect
	golang.org/x/mod v0.13.0 // indirect
	golang.org/x/net v0.18.0 // indirect
	golang.org/x/sync v0.5.0 // indirect
	golang.org/x/sys v0.14.0 // indirect
	golang.org/x/time v0.3.0 // indirect
	golang.org/x/tools v0.14.0 // indirect
	golang.org/x/xerrors v0.0.0-20220907171357-04be3eba64a2 // indirect
	google.golang.org/appengine v1.6.8 // indirect
	google.golang.org/appengine/v2 v2.0.2 // indirect
	google.golang.org/genproto v0.0.0-20231016165738-49dd2c1f3d0b // indirect
	google.golang.org/genproto/googleapis/api v0.0.0-20231016165738-49dd2c1f3d0b // indirect
	google.golang.org/genproto/googleapis/rpc v0.0.0-20231030173426-d783a09b4405 // indirect
	google.golang.org/grpc v1.59.0 // indirect
	google.golang.org/protobuf v1.31.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	inet.af/netaddr v0.0.0-20230525184311-b8eac61e914a // indirect
	moul.io/http2curl/v2 v2.3.0 // indirect
)

Compiler and Version used

go1.21.4 linux/arm64

Operating System and version

Debian GNU/Linux 12 (bookworm)

@khorii-cover khorii-cover added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Nov 16, 2023
@KaeDig
Copy link

KaeDig commented Nov 16, 2023

We've also noticed this bug in many of our modules in our dev environments since the sdk update, thank you for raising this issue.

Edit: And it's not just affecting sqs. Many services seem to be returning this error, in particular on list/get operations which we've encountered in our integration tests. Our modules which worked fine before are failing for example when testing if specific Codebuild projects exist using codebuild.BatchGetProjects, returning the "ResolveEndpointV2" error.

@khorii-cover
Copy link
Author

khorii-cover commented Nov 16, 2023

I just read the Release Notes.
I think this breaking change is likely the cause.

Feature: BREAKING CHANGE: V2 endpoint resolution middleware has changed steps from Serialize to Finalize. Middleware that indexes off of this field will need to be updated accordingly.

@khorii-cover
Copy link
Author

I think this line might possibly be where the error is occurring.

https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217

@khorii-cover
Copy link
Author

It seems like an error is occurring because ResolveEndpointV2 is not in the array, but it has been successfuly added at this point.

https://github.com/aws/aws-sdk-go-v2/blob/main/service/sqs/api_client.go#L139

@DeeptimanQlik
Copy link

DeeptimanQlik commented Nov 16, 2023

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
		aro.RoleSessionName = "RoleSession"
		aro.RoleARN =  role
	})

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}

@khorii-cover
Copy link
Author

I think this line might possibly be where the error is occurring.

https://github.com/aws/smithy-go/blob/main/middleware/ordered_group.go#L217

My earlier statement may have been incorrect.
I am looking at this section of the code now.

https://github.com/aws/smithy-go/blob/a2a3b0b7412a5b13cbadeae4f7349e3d1a4c37c2/middleware/ordered_group.go#L191

@khorii-cover
Copy link
Author

We are also facing same issue while initialising the aws.NewCredentialsCache(creds) and trying to retrieve the creds from the cache.

role := "arn:aws:iam::1222233333:role/sample-assume-role"
creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
		aro.RoleSessionName = "RoleSession"
		aro.RoleARN =  role
	})

credCache := aws.NewCredentialsCache(creds)
credentials, err := credCache.Retrieve(ctx)
if err != nil {
   // getting error as "failed to refresh cached credentials, not found, ResolveEndpointV2"

    return aws.Credentials{}, err 
}

I think it's because the same addProtocolFinalizerMiddlewares function is being called internally.

result, metadata, err := c.invokeOperation(ctx, "AssumeRole", params, optFns, c.addOperationAssumeRoleMiddlewares)

@lucix-aws lucix-aws self-assigned this Nov 16, 2023
@lucix-aws lucix-aws added p0 This issue is the highest priority and removed needs-triage This issue or PR still needs to be triaged. labels Nov 16, 2023
@lucix-aws
Copy link
Contributor

As you've called out from the release notes, we moved endpoint resolution from the Serialize step to Finalize. Per the notes, if you have calling code that was previously indexing off of that middleware (e.g. stack.Serialize.Insert(..., "ResolveEndpointV2", middleware.Before) it will have to be adjusted accordingly.

That said, I can't reproduce this, at least not minimally. This snippet runs without hitting the middleware insert error:

func main() {
    cfg, _ := config.LoadDefaultConfig(context.Background())
    svc := sts.NewFromConfig(cfg, func(o *sts.Options) {
        o.Region = "us-east-1"
    })
    if _, err := svc.AssumeRole(context.Background(), &sts.AssumeRoleInput{
        RoleArn:         aws.String("arn:aws:iam::1222233333:role/sample-assume-role"),
        RoleSessionName: aws.String("bar"),
    }); err != nil {
        panic(err)
    }
}

What I'm not seeing in either of your examples is how your client is constructed. Are you doing anything to modify APIOptions on the client? (or perhaps using some 1st or 3rd-party API that does so) Would you be able to provide the snippet for how your service clients are instantiated?

That aside I'm currently checking to see if we've missed anything that needed updated in our own APIs. The default path appears to be sound (our integration tests are all passing, this wouldn't have even been able to release if they hadn't), my current guess is we have some helper API somewhere that's still expecting ResolveEndpointV2 to be in Serialize.

@DeeptimanQlik
Copy link

DeeptimanQlik commented Nov 16, 2023

As you've called out from the release notes, we moved endpoint resolution from the Serialize step to Finalize. Per the notes, if you have calling code that was previously indexing off of that middleware (e.g. stack.Serialize.Insert(..., "ResolveEndpointV2", middleware.Before) it will have to be adjusted accordingly.

That said, I can't reproduce this, at least not minimally. This snippet runs without hitting the middleware insert error:

func main() {
    cfg, _ := config.LoadDefaultConfig(context.Background())
    svc := sts.NewFromConfig(cfg, func(o *sts.Options) {
        o.Region = "us-east-1"
    })
    if _, err := svc.AssumeRole(context.Background(), &sts.AssumeRoleInput{
        RoleArn:         aws.String("arn:aws:iam::1222233333:role/sample-assume-role"),
        RoleSessionName: aws.String("bar"),
    }); err != nil {
        panic(err)
    }
}

What I'm not seeing in either of your examples is how your client is constructed. Are you doing anything to modify APIOptions on the client? (or perhaps using some 1st or 3rd-party API that does so) Would you be able to provide the snippet for how your service clients are instantiated?

That aside I'm currently checking to see if we've missed anything that needed updated in our own APIs. The default path appears to be sound (our integration tests are all passing, this wouldn't have even been able to release if they hadn't), my current guess is we have some helper API somewhere that's still expecting ResolveEndpointV2 to be in Serialize.

@lucix-aws The AWS client config is initialised with region & request retryer.

please check here full code-snippet

package main

import (
	"context"
	"fmt"
	"time"
	"io"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/aws/retry"
	awsCfg "github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/kms"
	"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
	"github.com/aws/aws-sdk-go-v2/service/kms/types"
	"github.com/aws/aws-sdk-go-v2/service/sts"
)

func main() {

	ctx := context.Background()
	region := "us-east-1"

	kmsCfg, err := awsCfg.LoadDefaultConfig(ctx,
		awsCfg.WithRetryer(func() aws.Retryer {
			return retry.AddWithMaxAttempts(retry.NewStandard(), 10)
		}),

		awsCfg.WithRetryer(func() aws.Retryer {
			return retry.AddWithMaxBackoffDelay(retry.NewStandard(), time.Duration(10))
		}),
		awsCfg.WithRegion(region),
	)
	if err != nil {
		panic(err)
	}

	stsSvc := sts.NewFromConfig(kmsCfg)
	role := "arn:aws:iam::1222233333:role/sample-assume-role"
	creds := stscreds.NewAssumeRoleProvider(stsSvc, role, func(aro *stscreds.AssumeRoleOptions) {
		aro.RoleSessionName = "RoleSession"
		aro.RoleARN = role
	})
	credCache := aws.NewCredentialsCache(creds)
	kmsCfg.Credentials = credCache

	credentials, err := credCache.Retrieve(ctx)
	if err != nil {
		panic(err)
	}

	fmt.Println(credentials.AccessKeyID)

	client := kms.NewFromConfig(kmsCfg)
	res, err := client.GenerateDataKey(ctx, &kms.GenerateDataKeyInput{
		KeyId:             aws.String("arn:aws:kms:us-east-1:1122222333:key/21231232132132bf3232443242ae"),
		KeySpec:           types.DataKeySpecAes256,
		EncryptionContext: map[string]string{"fake-user": "fake-value"},
	}, func(o *kms.Options) {
		o.RetryMaxAttempts = 5
		o.EndpointOptions.Logger = o.Logger
	})
	if err != nil && err != io.EOF {
		panic(err)
	}
	fmt.Println(&res.KeyId)
}

@ucirello
Copy link

@lucix-aws

func defaultConfig(_ *testing.T) aws.Config {
	return aws.Config{
		Region: "us-west-2",
		EndpointResolverWithOptions: aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
			return aws.Endpoint{URL: "http://localhost:8000/"}, nil
		}),
		Credentials: credentials.StaticCredentialsProvider{
			Value: aws.Credentials{
				AccessKeyID:     "fakeMyKeyId",
				SecretAccessKey: "fakeSecretAccessKey",
			},
		},
	}
}
...
svc := dynamodb.NewFromConfig(defaultConfig(t))
...
_, err := c.dynamoDB.PutItem(ctx, putItemRequest)
	if err != nil {
		return nil, parseDynamoDBError(err, "cannot store lock item: lock already acquired by other client")
	}

fails for me.
You can refer to source and tests failing at: cirello-io/dynamolock#303

@lucix-aws
Copy link
Contributor

lucix-aws commented Nov 16, 2023

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue--

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday's release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello's package:

This combination of versions is broken:

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)

This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)

More detailed explanation to follow.

@lucix-aws lucix-aws added compatibility Issues with SDK, Go language version, or platform compatibility. and removed p0 This issue is the highest priority bug This issue is a bug. labels Nov 16, 2023
@phuongdnguyen
Copy link

phuongdnguyen commented Nov 16, 2023

To @ucirello , @DeeptimanQlik , @khorii-cover , @KaeDig and anyone else facing this issue--

Please upgrade ALL of your dependencies under the github.com/aws/aws-sdk-go-v2 namespace to the latest tags from yesterday's release, that should resolve the issue. Please chime in with the results if you can.

It appears that due to the breaking changes around middleware reordering, service client modules tagged on or after yesterday are now fundamentally incompatible with runtime modules tagged before yesterday (and vice versa).

For example, in the case of @ucirello's package:

This combination of versions is broken:

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday, expects ResolveEndpointV2 to be in Finalize
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.1 // released before yesterday, ResolveEndpointV2 is in serialize
)

This combination is valid (I have verified that your tests under v2/ pass on my system after upgrading):

    github.com/aws/aws-sdk-go-v2 v1.23.0 // released yesterday
    github.com/aws/aws-sdk-go-v2/service/dynamodb v1.25.2 // released yesterday, ResolveEndpointV2 is now in Finalize
)

More detailed explanation to follow.

Thanks, it works for me with aws.NewCredentialsCache

@koreyGambill
Copy link

@lucix-aws this combination of dependencies is broken for me when using SSO

	github.com/aws/aws-sdk-go-v2 v1.23.0
	github.com/aws/aws-sdk-go-v2/config v1.25.1
	github.com/aws/aws-sdk-go-v2/service/ec2 v1.134.0

this code

	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithSharedConfigProfile(o.Profile),
	)
	if err != nil {
		return fmt.Errorf("error loading config: %w", err)
	}
	client := ec2.NewFromConfig(cfg)
	_, err = client.DescribeInstances(context.TODO(), &ec2.DescribeInstancesInput{})
	if err != nil {
		return fmt.Errorf("error describing instances: %w", err)
	}

gives this error

error describing instances: operation error EC2: DescribeInstances, get identity: get credentials: failed to refresh cached credentials, the SSO session has expired or is invalid: failed to read cached SSO token file, open ..<redacted>.. no such file or directory

@lucix-aws lucix-aws changed the title ResolveEndpointV2 Error Encountered in SQS SendMessage service client modules released on or after 11/15/23 are incompatible against runtime modules from before that date (and vice versa) Nov 16, 2023
@lucix-aws
Copy link
Contributor

lucix-aws commented Nov 16, 2023

@koreyGambill does downgrading config to 1.23.0 resolve that issue?

More specifically, I see you're using an SSO session config, if you're using an sso_start_url in the config file with a fragment in the URL that's currently broken due to #2369 (a fix for that is merged, to be released today)

@jkakar
Copy link

jkakar commented Nov 16, 2023

I hit this issue and got past it by upgrading the github.com/aws/aws-sdk-go-v2 packages to the latest version.

@DeeptimanQlik
Copy link

DeeptimanQlik commented Nov 16, 2023

@lucix-aws Upgrading all the pending chore updates into one branch solves the issues for me.

go get github.com/aws/aws-sdk-go-v2@v1.23.0
go get github.com/aws/aws-sdk-go-v2/service/kms@v1.26.2
go get github.com/aws/aws-sdk-go-v2/credentials@v1.16.1
go get github.com/aws/aws-sdk-go-v2/config@1.25.1

wolfeidau added a commit to wolfeidau/iceberg-go that referenced this issue Jan 16, 2024
This resolves issues with endpoint changes in the SDK, see aws/aws-sdk-go-v2#2370, and upgrades s3iofs which has been upgraded to improve performance, see wolfeidau/s3iofs#25.
nastra pushed a commit to apache/iceberg-go that referenced this issue Jan 16, 2024
This resolves issues with endpoint changes in the SDK, see aws/aws-sdk-go-v2#2370, and upgrades s3iofs which has been upgraded to improve performance, see wolfeidau/s3iofs#25.
adambezecny added a commit to HRSuperSport/hrnogomet-backend-kit that referenced this issue Jan 23, 2024
this is to resolve error not found, ResolveEndpointV2, see
aws/aws-sdk-go-v2#2370
tcnghia added a commit to tcnghia/policy-controller that referenced this issue Jan 23, 2024
Some of these packages are <=v1.20 and some are >=v1.24 making it
susceptible to aws/aws-sdk-go-v2#2370

Signed-off-by: Nghia Tran <[email protected]>
@renanqts
Copy link

renanqts commented Jan 26, 2024

I also faced this issue in 1.24.1. I had to rollback to 1.23.0 to make it work

g-gaston added a commit to g-gaston/eks-anywhere that referenced this issue Jan 29, 2024
This should resolve an issue with incompatible aws sdk modules. This is
due to a recent update we did where we bumped some of them but not all.
Normally this should be fine, however, a change in the aws sdk made old
modules incompatible with new modules.

More info: aws/aws-sdk-go-v2#2370 (comment)
eks-distro-bot pushed a commit to aws/eks-anywhere that referenced this issue Jan 30, 2024
This should resolve an issue with incompatible aws sdk modules. This is
due to a recent update we did where we bumped some of them but not all.
Normally this should be fine, however, a change in the aws sdk made old
modules incompatible with new modules.

More info: aws/aws-sdk-go-v2#2370 (comment)
Vadman97 added a commit to highlight/highlight that referenced this issue Feb 3, 2024
## Summary

* Resolves AWS Marketplace callback via FormBody
`x-amzn-marketplace-token`
* Add frontend workspace picker for the AWS Marketplace callback to
connect the subscription.
* Fix aws SDK error on push payload
aws/aws-sdk-go-v2#2370

Tested to work with signed out user flow.

## How did you test this change?

![Screenshot from 2024-02-02
14-19-04](https://github.com/highlight/highlight/assets/1351531/a3331d4a-875b-4d05-9d91-656c110a04e0)

https://www.loom.com/share/3a2e537ba02c42d98dccf3eeabbe81dc
https://www.loom.com/share/c5540e1db6d0438b93476c43228c4de7
https://www.loom.com/share/185b00363bf049a19782c46744af9701
https://www.loom.com/share/e7af49f9b4df416b8e73aeb3a29652b1

## Are there any deployment considerations?

No

## Does this work require review from our design team?

No
@HemantDubey-ACS
Copy link

HemantDubey-ACS commented Feb 19, 2024

I'm still facing this issue while reading the secret manager

	cfg, _ := config.LoadDefaultConfig(context.TODO(), config.WithRegion("ap-south-1"))
	svc := secretsmanager.NewFromConfig(cfg)
	secrets, err := svc.GetSecretValue(context.TODO(), &secretsmanager.GetSecretValueInput{
		SecretId:     &secretId,
		VersionStage: aws.String("AWSCURRENT"),
	})
	if err != nil {
		fmt.Println("SECRET_STRING ", err.Error())	
	}

Expected Behavior
Retrieve the secret manager without errors as same is working on github.com/aws/aws-sdk-go-v2 v1.21.2

Current Behavior
When executing the code, the following error occurs:

not found, ResolveEndpointV2

Compiler and Version used
go1.21.1 darwin/arm64

Operating System and version
ProductName: macOS
ProductVersion: 14.2.1
BuildVersion: 23C71

AWS Go SDK V2 Module Versions Used

module github.com/AsthaBhopal/authentication

go 1.21

require (
	github.com/AsthaBhopal/pkgGoAsthaKycSchema v0.1.73
	github.com/AsthaBhopal/pkgGoAsthaLogs v0.9.0
	github.com/AsthaBhopal/pkgGoClevertapEvents v0.1.5
	github.com/AsthaBhopal/pkgGoFlowModels v1.0.63
	github.com/AsthaBhopal/pkgGoRedisAuthenticator v1.0.9
	github.com/aws/aws-sdk-go-v2 v1.25.0
	github.com/aws/aws-sdk-go-v2/config v1.18.22
	github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue v1.10.39
	github.com/aws/aws-sdk-go-v2/service/apigatewaymanagementapi v1.12.5
	github.com/aws/aws-sdk-go-v2/service/cognitoidentityprovider v1.22.6
	github.com/aws/aws-sdk-go-v2/service/dynamodb v1.21.5
	github.com/aws/aws-sdk-go-v2/service/lambda v1.39.5
	github.com/aws/aws-sdk-go-v2/service/secretsmanager v1.21.3
	github.com/aws/aws-sdk-go-v2/service/ses v1.16.10
	github.com/dgrijalva/jwt-go v3.2.0+incompatible
	github.com/gin-contrib/cors v1.4.0
	github.com/gin-contrib/pprof v1.4.0
	github.com/gin-gonic/gin v1.9.1
	github.com/gofrs/uuid v4.4.0+incompatible
	github.com/golang-jwt/jwt v3.2.2+incompatible
	github.com/joho/godotenv v1.5.1
	github.com/lib/pq v1.10.9
	github.com/nats-io/nats.go v1.32.0
	github.com/pquerna/otp v1.4.0
	github.com/yeqown/go-qrcode v1.5.10
	go.uber.org/zap v1.25.0
	golang.org/x/crypto v0.18.0
	gorm.io/driver/mysql v1.5.1
	gorm.io/driver/sqlserver v1.5.1
	gorm.io/gorm v1.25.4
	gorm.io/plugin/dbresolver v1.4.7
)

require (
	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.4.13 // indirect
	github.com/aws/aws-sdk-go-v2/credentials v1.13.36 // indirect
	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.11 // indirect
	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.0 // indirect
	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.0 // indirect
	github.com/aws/aws-sdk-go-v2/internal/ini v1.3.42 // indirect
	github.com/aws/aws-sdk-go-v2/internal/v4a v1.0.25 // indirect
	github.com/aws/aws-sdk-go-v2/service/apigateway v1.18.0 // indirect
	github.com/aws/aws-sdk-go-v2/service/dynamodbstreams v1.15.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/eventbridge v1.17.0 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.9.14 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery v1.7.35 // indirect
	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.35 // indirect
	github.com/aws/aws-sdk-go-v2/service/sso v1.13.6 // indirect
	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.15.5 // indirect
	github.com/aws/aws-sdk-go-v2/service/sts v1.21.5 // indirect
	github.com/aws/smithy-go v1.20.0 // indirect
	github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
	github.com/bytedance/sonic v1.10.2 // indirect
	github.com/cespare/xxhash/v2 v2.1.2 // indirect
	github.com/chenzhuoyu/base64x v0.0.0-20230717121745-296ad89f973d // indirect
	github.com/chenzhuoyu/iasm v0.9.1 // indirect
	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
	github.com/fogleman/gg v1.3.0 // indirect
	github.com/gabriel-vasile/mimetype v1.4.3 // indirect
	github.com/gin-contrib/sse v0.1.0 // indirect
	github.com/go-errors/errors v1.4.2 // indirect
	github.com/go-playground/locales v0.14.1 // indirect
	github.com/go-playground/universal-translator v0.18.1 // indirect
	github.com/go-playground/validator/v10 v10.16.0 // indirect
	github.com/go-redis/redis/v8 v8.11.5 // indirect
	github.com/go-sql-driver/mysql v1.7.0 // indirect
	github.com/goccy/go-json v0.10.2 // indirect
	github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
	github.com/golang-sql/sqlexp v0.1.0 // indirect
	github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/jackc/pgpassfile v1.0.0 // indirect
	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
	github.com/jackc/pgx/v5 v5.3.1 // indirect
	github.com/jinzhu/inflection v1.0.0 // indirect
	github.com/jinzhu/now v1.1.5 // indirect
	github.com/jmespath/go-jmespath v0.4.0 // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/klauspost/compress v1.17.2 // indirect
	github.com/klauspost/cpuid/v2 v2.2.6 // indirect
	github.com/leodido/go-urn v1.2.4 // indirect
	github.com/mattn/go-isatty v0.0.20 // indirect
	github.com/microsoft/go-mssqldb v1.1.0 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/nats-io/nkeys v0.4.7 // indirect
	github.com/nats-io/nuid v1.0.1 // indirect
	github.com/pelletier/go-toml/v2 v2.1.1 // indirect
	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
	github.com/ugorji/go/codec v1.2.12 // indirect
	github.com/yeqown/reedsolomon v1.0.0 // indirect
	go.uber.org/multierr v1.10.0 // indirect
	golang.org/x/arch v0.6.0 // indirect
	golang.org/x/image v0.0.0-20200927104501-e162460cd6b5 // indirect
	golang.org/x/net v0.19.0 // indirect
	golang.org/x/sys v0.16.0 // indirect
	golang.org/x/text v0.14.0 // indirect
	google.golang.org/protobuf v1.32.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	gorm.io/driver/postgres v1.5.2 // indirect
)

@lucix-aws
Copy link
Contributor

lucix-aws commented Feb 20, 2024

All--

Since we're still getting comments on this issue, I think the diagnosis/solution has been somewhat buried.

To reiterate: this error occurs when using a version of the core aws-sdk-go-v2 module at or above v1.23.0 (essentially, released on or after 11/15/23) in combination with a service module released before that date. The most recent comment from @HemantDubey-ACS is an example of this - the service/secretsmanager v1.21.3 is from before that breaking point.

Upgrading all modules under the aws-sdk-go-v2 namespace will resolve this issue. You can do so using the following command (TYVM @droberts-sea):

go get -u github.com/aws/aws-sdk-go-v2/...

This was originally diagnosed here, and you can read a post-mortem on this issue in this comment.

To be absolutely clear, the bug users experienced here was NOT intentional, and CAN be avoided in the future independent of any middleware reordering we do. This incident has greatly informed how we should be writing/managing the code responsible for building the operation middleware stack. I've spawned #2507, the addressing of which should eliminate the possibility for breaks like this to occur in the future.

To guarantee the visibility of this message, I'm going to lock this thread. If you are still seeing this issue after updating all of your SDK modules, please open a new issue.

@aws aws locked and limited conversation to collaborators Feb 20, 2024
@lucix-aws lucix-aws changed the title [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules released on or after 11/15/23 are incompatible against previous runtimes (and vice versa) [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules released on or after 11/15/23 are incompatible against previous runtimes Feb 20, 2024
@lucix-aws lucix-aws changed the title [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules released on or after 11/15/23 are incompatible against previous runtimes [SOLUTION IN THREAD] "not found, ResolveEndpointV2" service modules before 11/15/23 are incompatible against newer root modules Feb 20, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
compatibility Issues with SDK, Go language version, or platform compatibility.
Projects
None yet
Development

No branches or pull requests