Improving docs to support configuration questions

[mtliendo]

First timer with SSO here 👋🏽 After following this guide on configuring SSO, and then taking Ben's advice for setting up CLI access, I have the following simple questions and I'd be happy to PR the docs so that it can help others. Personally, I think some of my stumbling blocks stem from having to switch my mental model from thinking in accounts to thinking in roles/SSO.

A few of the questions are based off of this code snippet in the readme:

[profile my-sso-profile]
sso_start_url = https://example.awsapps.com/start
sso_region = us-east-1 # the region Identity Center is configured in
sso_account_id = 123456789012
sso_role_name = MyRoleName
region = us-east-2 # the region to use for AWS API calls

1. Which parts of this refer to the account that I configured Identity Center in and which refers to the account I want to use?
2. If I leave this empty, and run aws-sso-util configure profile [my-profile] I get "No Identity Center instance found". If I update the config above and set the sso_account_id to one of my org accounts, then I get a credential_process key added. How do I get the CLI to prompt me to enter in values?
3. When testing to see if this works, I ran aws-sso-util login. Is this needed to provide temporary credentials--thus removing the need for an ~/.aws/credentials file?
4. After logging in, how do I run AWS CLI commands? Doing aws s3 ls yields an "Unable to locate credentials. You can configure credentials by running "aws configure" error. Passing in the profile with --profile gave me returned "An error occurred (ForbiddenException) when calling the GetRoleCredentials operation: No access" (this user has the the powerUser role applied).

Again, happy to update the docs to provide guidance for others as well!

[mtliendo]

Update since I was running this during a us-east-1 outage 😓

1. If there is already a config as shown above, then the CLI with prompt as normal when running aws-sso-util configure profile will create a new one as expected, otherwise I haven't gotten it to work.
2. sso_account_id seems to refer to the account I want to authenticate as
3. the profile name is a local identifier as in a non-sso environment (probably best to name it the same as the aws account it's representing, but it doesn't have to be(?)
4. I had the role wrong. I had PowerUserManagement but that actual name is PowerUserAccess. Careful when copying values folks! If the role is misspelled then you'll get an error. If the role is correct, but there's an outage in that region you'll get a 500 max retries exceeded error 😅
5. After logging in, you use the regular AWS CLI but pass in the profile you're using. I read in this repos docs something about exporting variables so that's not the case or just naming it default, but I think I'll leave it as is for now.

How does all that sound? Any red flags in my setup? If not, there are some areas I'd like to update in the docs :)

[benkehoe]

Ok, let's see. Trying to do this after a long day at re:Inforce, so let me know if any of this makes sense.

A "profile" is a set of configuration values for the CLI and SDKs, including credentials OR how to get credentials. The values are taken from ~/.aws/config and ~/.aws/credentials, the latter being for storing static credentials, which doesn't apply here. For Identity Center, getting credentials for a particular account and role requires four pieces of information. The first two identify the Identity Center instance itself, and these are the start URL (the sso_start_url field, usually something like https://example.awsapps.com/start, this is also the URL you use to log in to the console) and the region that Identity Center is configured in (so that the CLI/SDK knows what regional endpoint to talk to Identity Center in), this is the sso_region field. These two are sufficient to log in, because you don't "log in" to an account and role, you log in to Identity Center and get a session that can be used to access all your accounts and roles. The second two are the account and role, sso_account_id and sso_role_name, respectively. Note that the region in which the credentials are actually used for a profile is a separate field region, which does not have to be the same as the region where Identity Center is configured.

The old style of profile configuration puts all four of these in each profile, and that's what aws-sso-util configure currently supports. There's a new style which puts the start URL and Identity Center region in a special ~/.aws/config section called sso-session, and then in a profile instead of putting those there, you use the sso_session field to reference the session you've defined. You can read about that here: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html I've been meaning to update aws-sso-util configure to support this.

Now, generally aws-sso-util needs to know your start URL and region, but it tries to be smart and find them for you. So it looks in the profiles in ~/.aws/config and if all of them use the same, it'll just use that. Normally if it doesn't find any, say for aws-sso-util login, that's an error. But it shouldn't be for aws-sso-util configure profile, that should result in an interactive prompt. Can you open a separate bug for that?

The docs detail how that search is performed, for example here: https://github.com/benkehoe/aws-sso-util/blob/master/docs/login.md#identity-center-instances So once you have something it can find (or provide it directly on the command line), it'll do its thing.

You can do aws-sso-util configure profile to configure a single profile, but in general I'd recommend using aws-sso-util configure populate, for which the docs are here. It will make sure you're logged in and then create profiles for every account and role you have (and even for multiple regions if that's important to you) with standardized names based on the account and role. There's a lot of configurability for those profile names if you want it, but the point is, then you don't need to create profiles directly, you'll just know what the one you want will be named and you can just use it. I'd also recommend explicitly setting the start URL and region in the command, just so you know what you're getting.

As I said, aws-sso-util login logs you in Identity Center. It results in a token that's cached in a file under ~/.aws/sso/cache. That token can be used (by the CLI/SDKs automatically) to get credentials for any account/role you have access to. So when logging in, you do that once, not for each account and role you might be using.

Now, when you're using the CLI or SDK, it has a search mechanism for finding credentials. That is documented here, but the point is it looks in lots of places. For this, we care about telling it to use one of the profiles we've configured. That can be done with the CLI with the --profile command (and if you're a tool builder, I recommend providing that flag for your users as well), or for the CLI or SDKs (so any tool using an AWS SDK will work this way) with the AWS_PROFILE environment variable. I often use this method and I wrote a shell function to help manage it, you can see that here.

When the CLI or SDK is given a profile name, it looks up the config for that profile, which (hopefully) tells it how to get credentials (or provides static creds in ~/.aws/credentials, but hopefully not!). For Identity Center, that's those four values we spoke of. It uses the start URL and Identity Center region to look up that cached token, and the account and role to use that token with Identity Center to get the familiar access key id, secret access key, and session token for that account and role. Note that it can do this for any account and role you've got access to without needing a different token.

Now, as an aside, you can have a special profile named default, and if no profile is set and no other credentials are found, it'll use that profile. But I think that's bad! If you try to set credentials or a profile, but do it wrong, it will silently use the wrong credentials, and that could be bad! So I think it's better to not have a default profile, and then if you've done it wrong, it'll just give an error.

So overall I would recommend set up that looks something like this:

$ aws-sso-util configure populate \
    --sso-start-url https://example.awsapps.com/start --sso-region us-east-2 \
    --region us-east-1

# example of what the config could look like
$ cat ~/.aws/config
[profile my-account-1.RoleOne]
sso_start_url = https://example.awsapps.com/start
sso_region = us-east-2
sso_account_id = 111122223333
sso_role_name = RoleOne
region = us-east-1

[profile my-account-1.RoleTwo]
sso_start_url = https://example.awsapps.com/start
sso_region = us-east-2
sso_account_id = 111122223333
sso_role_name = RoleOne
region = us-east-1

[profile my-account-2.RoleTwo]
sso_start_url = https://example.awsapps.com/start
sso_region = us-east-2
sso_account_id = 444455556666
sso_role_name = RoleTwo
region = us-east-1

$ aws-sso-util login
# searches your config and finds https://example.awsapps.com/start & us-east-2
# initiates browser pop up, you log in, command completes with token in cache
# now just use those profiles

$ aws sts get-caller-identity --profile my-account-1.RoleOne
Arn: arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_RoleOne_abc123/ben

$ aws sts get-caller-identity --profile my-account-1.RoleTwo
Arn: arn:aws:sts::111122223333:assumed-role/AWSReservedSSO_RoleTwo_def456/ben

$ aws sts get-caller-identity --profile my-account-2.RoleTwo
Arn: arn:aws:sts::444455556666:assumed-role/AWSReservedSSO_RoleTwo_ghi789/ben

[ravenium]

There's a new style which puts the start URL and Identity Center region in a special ~/.aws/config section called sso-session, and then in a profile instead of putting those there, you use the sso_session field to reference the session you've defined. You can read about that here: https://docs.aws.amazon.com/cli/latest/userguide/sso-configure-profile-token.html I've been meaning to update aws-sso-util configure to support this.

Yes please! Not that the current config doesn't work well (you basically helped eliminate a lot of gruntwork in "please write out these 40 profiles" for users, so thank you!), but the "new" way is a bit more elegant. Also, is it necessary to have credential_process here? This feels like somewhat of a tautology if one is using something like aws-vault.

[jwdinkel]

Glad I found this. The readme didn't suggest this command and it seems to have a problem for me.
Seems we should put this example on the section here: https://github.com/benkehoe/aws-sso-util/tree/master#configuring-awsconfig

Maybe somebody can be kind enough to assist with making it work, however...
It's not a big deal because I can still login using the configure profile, but configure populate fails, perhaps because of the number of accounts. Perhaps there is a way to batch out those requests.

I have 150 accounts and I attempted to run this using a clean (Ephemeral) Google cloud shell
https://shell.cloud.google.com/?show=ide%2Cterminal

This command will let you edit the config file if needed

cloudshell edit ~/.aws/config

Shell results with personal info removed:

jason@cloudshell:~$ pipx install aws-sso-util
installed package aws-sso-util 4.32.0, installed using Python 3.9.2
These apps are now globally available
- aws-sso-util
done! ✨ 🌟 ✨

jason@cloudshell:~$ aws-sso-util configure populate
--sso-start-url https://example.awsapps.com/start# --sso-region eu-west-1
--region eu-west-1
Logging in to https://example.awsapps.com/start#
Login with IAM Identity Center required.
Attempting to open the authorization page in your default browser.
If the browser does not open or you wish to use a different device to
authorize this request, open the following URL:

https://device.sso.eu-west-1.amazonaws.com/

Then enter the code:

WZMD-KMFD

https://device.sso.eu-west-1.amazonaws.com/?user_code=WZMD-KMFD
Gathering accounts and roles
Traceback (most recent call last):
File "/home/jason/.local/bin/aws-sso-util", line 8, in
sys.exit(cli())
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/aws_sso_util/populate_profiles.py", line 342, in populate_profiles
response = client.list_account_roles(**list_role_args)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/jason/.local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the ListAccountRoles operation (reached max retries: 4): HTTP 429 Unknown Code

The problem seems to be related to https://github.com/benkehoe/aws-sso-util/blob/master/cli/src/aws_sso_util/populate_profiles.py

Cloudshell can be connected to your local environment dev tools like this: https://cloud.google.com/shell/docs/using-cloud-shell-with-gcloud-cli

Otherwise we can edit the python code in place by executing:

cloudshell edit .local/pipx/venvs/aws-sso-util/lib/python3.9/site-packages/aws_sso_util/populate_profiles.py

I'll come back to this if I can help further.

[iainelder]

@jwdinkel , try setting environment variables to make the SDK keep trying.

Same error reported here: An error occurred (TooManyRequestsException) when calling the ListAccountRoles operation

#112

OP confirmed the solution was to set these before running configure populate:

export AWS_RETRY_MODE=standard
export AWS_MAX_ATTEMPTS=100

Just out of interest, if you can tell me, how many profiles do you expect to generate? As you point out the error seems to be more likely in bigger AWS estates.