service/cognitoidentityprovider: Add Paginators

[the1337beauty]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

v1.10.7

Version of Go (go version)?

v1.5.3

What issue did you see?

Pagination functions for Cognito Identity Provider do not exist event though some cognito-idp requests return a pagination token (ex. ListUsers). I'd like a separate function similar to S3's ListObjectsPages or some method to loop through cognito's paginated actions to get all results.

Steps to reproduce

Unable to provide exact results since functions do not exist.
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/making-requests.html#using-pagination-methods
The above example cannot be replicated because a similar function does not exist.

If you have have an runnable example, please include it.
This is the only example I've been able to get running but it returns a single set of results with the pagination token so I still have to manually check the pagination token and loop through to get all results which I haven't gotten working yet.

func cogListUsers() (*cognitoidentityprovider.ListUsersOutput, error) {
	svc := cognitoidentityprovider.New(sess)
	attr := []string{"email"}
	params := &cognitoidentityprovider.ListUsersInput{
		AttributesToGet: aws.StringSlice(attr),
		UserPoolId:      aws.String(UserPoolID),
	}
	result, err := svc.ListUsers(params)
	if err != nil {
		aerr, _ := err.(awserr.Error)
		log.Println("error: ", aerr)
	}
	log.Println("ListUsers result: ", result)
	return result, err
}

[xibz]

Hello @the1337beauty, thank you for reaching out to us. We've went ahead and marked this as a feature request. If you find anymore of these, please let us know!

[the1337beauty]

@xibz thank you, I had a call yesterday with some AWS folks to discuss my experience with Cognito and I brought this up in that call.

[the1337beauty]

Is there any update on this? Will it be implemented in Go SDK v2? @jasdel ?

[BastienM]

Hi there,

A little bump on the subject.

I am using the snippet found here to "iterate" through the pages returned by cognitoidentityprovider.ListUsers(), sadly it only browse one page before failing silently ...
The pool has over 600+ users and query 50 items per page.

Go version

go version go1.10 darwin/amd64

SDK

version: ~1.13.3

Code extract

func getUsersFromPool(pool *cognitoidentityprovider.UserPoolDescriptionType) (*cognitoidentityprovider.ListUsersOutput, error) {
	var users []*cognitoidentityprovider.UserType

	params := cognitoidentityprovider.ListUsersInput{
		UserPoolId: &*pool.Id,
		Limit:      aws.Int64(50),
	}
	ctx := context.Background()

	p := request.Pagination{
		NewRequest: func() (*request.Request, error) {
			req, _ := CognitoClient.ListUsersRequest(&params)
			req.SetContext(ctx)
			return req, nil
		},
	}

	for p.Next() {
		page := p.Page().(*cognitoidentityprovider.ListUsersOutput)
		for _, obj := range page.Users {
			users = append(users, obj)
		}
	}	

	output := &cognitoidentityprovider.ListUsersOutput{}
	output.SetUsers(users)

	return output, p.Err()
}

Is something missing ?

@the1337beauty I think this method is what you were looking for, no ?

[the1337beauty]

I had found another way to paginate but thank you for reaching out @BastienM

[jasdel]

Thanks for the update @the1337beauty and @BastienM it looks like the CognitoIdentityProvider service does not define paginators for their service in the models provided by them to the SDKs. This lack of information is why the request.Pagination type isn't working. The request.Pagination type requires additional information to know how to paginate the APIs, which in the case of this service is not modeled.

I've reached out to the service team asking them to add information to their model that provides pagination information.

The following is an workaround you can use which will add pagination metadata to the ListUsers API so you can use the SDK's provided request.Pagination utility. The example adds a NewPageableListUsersRequest helper which augments the request returned by ListUsersRequest to include pagination metadata. The getUsersFromPool function was also updated to use this helper.

func getUsersFromPool(pool *cognitoidentityprovider.UserPoolDescriptionType) (*cognitoidentityprovider.ListUsersOutput, error) {
	var users []*cognitoidentityprovider.UserType

	params := cognitoidentityprovider.ListUsersInput{
		UserPoolId: &*pool.Id,
		Limit:      aws.Int64(50),
	}
	ctx := context.Background()

	p := request.Pagination{
		NewRequest: func() (*request.Request, error) {
			req := NewPageableListUsersRequest(CognitoClient, &params)
			req.SetContext(ctx)
			return req, nil
		},
	}

	for p.Next() {
		page := p.Page().(*cognitoidentityprovider.ListUsersOutput)
		users = append(users, page.Users...)
	}

	output := &cognitoidentityprovider.ListUsersOutput{}
	output.SetUsers(users)

	return output, p.Err()
}

// NewPageableListUsersRequest returns a new AWS SDK request constructed for the ListUsers API which can be paged over.
func NewPageableListUsersRequest(svc cognitoidentityprovideriface.CognitoIdentityProviderAPI, in *cognitoidentityprovider.ListUsersInput) *request.Request {
	req, _ := svc.ListUsersRequest(in)
	if req.Operation.Paginator != nil {
		return req
	}

	req.Operation.Paginator = &request.Paginator{
		InputTokens:  []string{"PaginationToken"},
		OutputTokens: []string{"PaginationToken"},
		LimitToken:   "Limit",
	}

	return req
}

[BastienM]

Hi @jasdel,

Thank you for the code snippet, very much appreciated.
I can now iterate over all the users without further errors.

I also notified the person in charge of the ticket I opened to AWS Support of your answer in order to speed things up.

[tomelliff]

Am I missing something about how the various SDKs get generated?

Both this SDK and botocore seem to independently add models including the paginator JSON for each service rather than be something that is automatically generated/provided upstream and then the related paginators generated from that.

Over at botocore, both boto/botocore#1462 and boto/botocore#1633 refer to a script that one of the botocore team members wrote to generate the paginators in that pull request but it looks like that was a one off effort and the script isn't in the botocore repo that I can see or shown in the issue or the PR comments.

I'd personally like to see the ListIdentityPools paginator added here so that I can simplify some code I'm about to write for Terraform's AWS provider (getting into crazy levels of yak shaving now) and am happy to just copy across botocore's paginator JSON into models/apis/cognito-identity/2014-06-30/paginators-1.json but this feels like an overly manual and specific approach.

Edit: Looking at the PR template and that the aws-sdk-go-automation bot has been raising all the recent pagination changes then I guess that bot is meant to be picking this up but then it's not clear how it looks that up and why ListIdentityPools isn't available here when it is in botocore.

[github-actions]

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.