ec2.NewDescribeInstanceTypesPaginator response differs from AWS CLI response for the same filter

[ArcticSnowman]

Acknowledgements

• I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
• I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

When attempting to get descriptions of instance types with a filter the response from the GO SDK differs from the AWS CLI response.

Take the code snippet:

var filters []ec2types.Filter
var instanceType string = "instance-type"

filter := ec2types.Filter{
	Name:   &instanceType,
	Values: []string{iType},
}

filters = append(filters, filter)

desInstTypeParms := ec2.DescribeInstanceTypesInput{
	Filters: filters,
}
pager := ec2.NewDescribeInstanceTypesPaginator(ec2Client, &desInstTypeParms)

Where the iType is m6i.16xlarge , the GO SDK will get no match, while the AWS CLI will get a match

$ aws ec2 describe-instance-types --filter Name=instance-type,Values="m6i.16xlarge"  --query "InstanceTypes[*].InstanceType" --output text
m6i.16xlarge

Running DEBUG mode for both I noticed that the GO SDK sends the request body as application/x-www-form-urlencoded encoded.

The issue seems to be the full stop . in the filter value. For example is the value were m6i*xlarge the response would be the same.

I ran the --debug on the command line version but that does not output the raw request body, so I'm unable to determine what the differences are in the request to the AWS API.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would expect the GO SDK to return the same data that the AWS CLI does.

Current Behavior

GO SDK get no matches back

Reproduction Steps

package main

import (
	"context"
	"log"
	"os"

	"github.com/aws/aws-sdk-go-v2/aws"
	awsconfig "github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/ec2"
	ec2types "github.com/aws/aws-sdk-go-v2/service/ec2/types"
	"github.com/aws/smithy-go/logging"
)

func main() {

	awscfg, err := awsconfig.LoadDefaultConfig(context.Background(),
		awsconfig.WithSharedConfigProfile("aws-stg"),
		awsconfig.WithRegion("us-east-1"),
		awsconfig.WithLogger(logging.NewStandardLogger(os.Stderr)),
		awsconfig.WithClientLogMode(aws.LogResponse|aws.LogRequestWithBody|aws.LogRequestEventMessage))

	if err != nil {
		log.Fatal(err)
	}

	ec2Client := ec2.NewFromConfig(awscfg)

	var filters []ec2types.Filter
	var instanceType string = "instance-type"
	var iType = "m6i.16xlarge"

	filter := ec2types.Filter{
		Name:   &instanceType,
		Values: []string{iType},
	}

	filters = append(filters, filter)

	desInstTypeParms := ec2.DescribeInstanceTypesInput{
		Filters: filters,
	}

	pager := ec2.NewDescribeInstanceTypesPaginator(ec2Client, &desInstTypeParms)

	for pager.HasMorePages() {
		output, err := pager.NextPage(context.Background())
		if err != nil {
			log.Println(err)
			return
		}

		if len(output.InstanceTypes) == 0 {
			log.Printf("Unable to get details on instance types matching `%s`", iType)
			return
		}

		for _, t := range output.InstanceTypes {
			log.Printf("Type: %s", t.InstanceType)
		}
	}

}

Using this snippet of code, testing a few values for iType
m6i.16xlarge - Nothing found
m6i*16xlarge - Nothing Found
*16xlarge- Nothing found
m6i* - List found
In all cases the command line tool returns an instance-type

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

        github.com/aws/aws-sdk-go-v2 v1.32.5
        github.com/aws/aws-sdk-go-v2/config v1.28.5
        github.com/aws/aws-sdk-go-v2/service/autoscaling v1.51.0
        github.com/aws/aws-sdk-go-v2/service/ec2 v1.194.0
        github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.7 // indirect
        github.com/aws/aws-sdk-go-v2/credentials v1.17.46 // indirect
        github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.20 // indirect
        github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.24 // indirect
        github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.24 // indirect
        github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
        github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.24 // indirect
        github.com/aws/aws-sdk-go-v2/service/elasticloadbalancing v1.28.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.43.0 // indirect
        github.com/aws/aws-sdk-go-v2/service/eventbridge v1.35.6 // indirect
        github.com/aws/aws-sdk-go-v2/service/iam v1.38.1 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.1 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.4.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/kms v1.37.6 // indirect
        github.com/aws/aws-sdk-go-v2/service/pricing v1.32.6 // indirect
        github.com/aws/aws-sdk-go-v2/service/route53 v1.46.2 // indirect
        github.com/aws/aws-sdk-go-v2/service/s3 v1.69.0 // indirect
        github.com/aws/aws-sdk-go-v2/service/sqs v1.37.1 // indirect
        github.com/aws/aws-sdk-go-v2/service/ssm v1.56.0 // indirect
        github.com/aws/aws-sdk-go-v2/service/sso v1.24.6 // indirect
        github.com/aws/aws-sdk-go-v2/service/ssooidc v1.28.5 // indirect
        github.com/aws/aws-sdk-go-v2/service/sts v1.33.1 // indirect

Compiler and Version used

go version go1.23.2 linux/amd64

Operating System and version

Centos 7

[Madrigal]

The problem is that the filter on this operation is not "filter before you make the request" but rather "once you get the response, apply this filter". Hence, when you do this

		if len(output.InstanceTypes) == 0 {
			log.Printf("Unable to get details on instance types matching `%s`", iType)
			return
		}

You short-circuit if the response is not found on the first page. Worse, since the response is not guaranteed to maintain the same order, it may succeed one time and fail the next.

This behavior is sort-of implied on the docs, but I agree it could be made more explicit

If you specify filters, the output includes information for only those instances that meet the filter criteria.

If you get rid of that return statement, you'll see that all the cases you listed succeed

As from the other command

I ran the --debug on the command line version but that does not output the raw request body, so I'm unable to determine what the differences are in the request to the AWS API.

You do actually get the full HTTP request with --debug on the CLI. Look for output that has Sending http request, like this

2024-11-27 15:25:36,814 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://ec2.us-west-2.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.22.2 md/awscrt#0.22.0 ua/2.0 os/macos#24.1.0 md/arch#arm64 lang/python#3.12.7 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#ec2.describe-instance-types', 'X-Amz-Date': b'20241127T202536Z', 'X-Amz-Security-Token': b'XXXXX', 'Authorization': b'AWS4-HMAC-SHA256 Credential=XXXXXXX/20241127/us-west-2/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=XXXXX', 'Content-Length': '105'}>

Although since this shows the raw HTTP request with a signed request, so you'll probably find the log Making request for OperationModel more useful

2024-11-27 15:25:36,813 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeInstanceTypes) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.22.2 md/awscrt#0.22.0 ua/2.0 os/macos# lang/python#3.12.7 md/pyimpl#CPython cfg/retry-mode#standard md/installer#source md/prompt#off md/command#ec2.describe-instance-types'}, 'body': {'Action': 'DescribeInstanceTypes', 'Version': '2016-11-15', 'Filter.1.Name': 'instance-type', 'Filter.1.Value.1': 'm6i.16xlarge'}, 'url': 'https://ec2.us-west-2.amazonaws.com/', 'context': {'client_region': 'us-west-2', 'client_config': <botocore.config.Config object at 0x1043af440>, 'has_streaming_input': False, 'auth_type': None, 'unsigned_payload': None}}

You can see that this is not just one request, but rather around 9 requests until it gets a result.

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.

[ArcticSnowman]

@Madrigal - You are correct.

Here is the updated code showing the correct way to check for no instances found

		if len(output.InstanceTypes) == 0 {
			if output.NextToken == nil {
				if typesTotal == 0 {
					log.Printf("Unable to get details on instance types matching `%s`", iType)
					return
				} else {
					break
				}
			}
			continue
		}