in ecs got security token service credentials is empty.

[dybxin]

Documentation

• I've gone though the API reference
• I've checked AWS Forums and StackOverflow for answers
• I've searched for previous similar issues and didn't find any solution

Describe the bug

in ecs cluster, i deploy services in fargate. when i use aws-sdk-go-v2 to get security token service credentials, i got an empty response result.

the response result not contains AccessKeyID 、SecretAccessKey and SessionToken.

Expected behavior

i expected got the complete response.

Current behavior

2022-01-06T17:37:39.064+08:00 | value &{ AssumeRoleProvider false 0001-01-01 00:00:00 +0000 UTC}
2022-01-06T17:39:10.810+08:00 |
err operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: F3ARTKKHJ21SGJ1J, HostID: sH3s7sJOnyzMI63/VE6uFEmF9KYXKiRohUwqPaRre5DAIn/xCUCsBwRqCai6mRFc4FWGa9WFc04=, api error AuthorizationHeaderMalformed: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.

Steps to Reproduce

// the roleArn is the ecsTaskExecutionRole role's arn.
roleArn = "arn:aws-cn:iam::xxxxx:role/ecsTaskExecutionRole"
region  = "cn-northwest-1"

config, err := config.LoadDefaultConfig(context.Background(), config.WithRegion(region))
if err != nil {
	panic(error)
}

stsSvc := sts.NewFromConfig(config)
creds := stscreds.NewAssumeRoleProvider(stsSvc, roleArn)
value, err := creds.Retrieve(context.TODO())
if err != nil {
	// handle error
}
fmt.Println("value", &value)
config.Credentials = aws.NewCredentialsCache(creds)

s3Cient := s3.NewFromConfig(config)

output, err := s3Cient.CreateBucket(context.TODO(), &s3.CreateBucketInput{
	Bucket:                    aws.String("images"),
	CreateBucketConfiguration: &types.CreateBucketConfiguration{LocationConstraint: 
        types.BucketLocationConstraintCnNorthwest1},
})

if err != nil {
	fmt.Println("err", err)
	var alreadyOwnedByYou *types.BucketAlreadyOwnedByYou
	var alreadyExists *types.BucketAlreadyExists
	if errors.As(err, &alreadyOwnedByYou) || errors.As(err, &alreadyExists) {
		//return
		fmt.Println("------")
	}
}
fmt.Sprintf("bucket location: %s", aws.ToString(output.Location))

Possible Solution

No response

AWS Go SDK version used

1.16.6

Compiler and Version used

go version go1.16.6 linux/amd64

Operating System and version

Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type

[chrnorm]

I had some similar error messages using an outdated SDK version - they were solved for me by updating the SDK with go get github.com/aws/aws-sdk-go-v2. Not sure if this is helpful so I am watching this ticket too for any news on a breaking API change on AWS's side.

[dybxin]

I had some similar error messages using an outdated SDK version - they were solved for me by updating the SDK with go get github.com/aws/aws-sdk-go-v2. Not sure if this is helpful so I am watching this ticket too for any news on a breaking API change on AWS's side.

my problem is solved. in ecs fargate i not set public IP, caused by get sts svc timeout.

[KaibaLopez]

Hi @dybxin ,
Just to be clear, have you solved your issue or is this still a problem you are facing?

[dybxin]

Hi @dybxin , Just to be clear, have you solved your issue or is this still a problem you are facing?

yeah i solved already.

[udf2457]

@KaibaLopez @dybxin

I seem to be getting the same issue ? I think there might be a bug here ?

My code (based on the example given in the docs):

func AssumeRoleWithSts(roleARN string) (aws.Config, error) {
	cfg, err := config.LoadDefaultConfig(context.TODO())
	if err != nil {
		return aws.Config{}, err
	}
	stsSvc := sts.NewFromConfig(cfg)
	creds := stscreds.NewAssumeRoleProvider(stsSvc, roleARN)
	cfg.Credentials = aws.NewCredentialsCache(creds)
	return cfg, nil
}

Which is then passed into an S3 caller ...

func Upload(cfg aws.Config) error {
	client := s3.NewFromConfig(cfg)
etc. etc.

I get the following error:

operation error S3: ListObjectsV2, https response error StatusCode: 400, RequestID: HB4N3M8KNPXC6XTK, HostID: n3Y0V+t+0hUEz+gMGUF4Daeca2QMrDVnTjCVBOqRkhxhOTUuUSJTKqvz0XNHKKIQ68pRgRf3mjA=, api error AuthorizationHeaderMalformed: The authorization header is malformed; a non-empty Access Key (AKID) must be provided in the credential.

However, if I rely on the ~/.aws/credentials method instead, i.e.:

func LoadBasicAuthWithProfile(profile string) (aws.Config, error) {
	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithSharedConfigProfile(profile))
	if err != nil {
		return aws.Config{}, err
	}
	return cfg, nil
}

and defining role_arn and source_profile in there, it works.

Weird !

[oakad]

Recently, I managed to trigger this very error on EKS. Upon investigation, a culprit of my issue was as following:

1. Trust policy for the intended pod role had wrong service account value specified in the condition block.
2. This, in turn, caused "implicit" AssumeRoleWithWebIdentity call issued by the authentication provider to fail. This failure is not properly processed or reported by authentication provider, at least in my version of the SDK (1.16.14 base version and corresponding component versions).
3. All subsequent calls issues via the s3 client (v1.27.2) would then fail with the AKID missing error.

In short, web identity authentication provider should be modified to report errors properly and not just swallow them.

[RanVaknin]

Hi all,

It seems like there are various reasons for this, all of them are due to incorrect configurations with ECS / EKS.

@oakad , errors reported from the credential providers will always be hidden because the credential chain is meant to work implicitly. The default config will go through every provider in the chain in order, with each one failing until it finds the one that is able to return credentials successfully. If we were to log each failure to standard out, it will create logging noise. The way around it is to set the appropriate log level on your config object so that you may see the underlying failed outgoing requests.

You can do that by doing the following:

	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithRegion("us-east-1"),
		config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody), // enable request and response logging
	)

Since this issue was reported by a number of people, and was caused and mitigated by different reasons Im going to go ahead and close this.
If this is still a problem, please create a new separate issue so we may better assists you one a per-customer cadence.

Thanks again,
Ran~

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.