tsh proxy aws fails after one minute when using per-session MFA

[webvictim]

Expected behavior: tsh app login <aws-console-app-name> should give you a certificate with a useful lifetime when using per-session MFA.

Current behavior: tsh proxy aws stops working after 1 minute, because the certificate issued is only valid for that long.

Logs:

# log into app, get prompted for MFA as expected
gus@apollo:~ % tsh app login awsconsole --aws-role PurpleAWSEC2FullAccess
MFA is required to access Application "awsconsole"
Tap any security key
Detected security key tap
Logged into AWS app "awsconsole".

Your IAM role:
  arn:aws:iam::<account>:role/PurpleAWSEC2FullAccess

Example AWS CLI command:
  tsh aws s3 ls

Or start a local proxy:
  tsh proxy aws --app awsconsole

# start local proxy in background
gus@apollo:~ % tsh proxy aws &
[1] 10617

gus@apollo:~ [fg: 1] % Started AWS proxy on http://127.0.0.1:54430.
To avoid port randomization, you can choose the listening port using the --port flag.

Use the following credentials and HTTPS proxy setting to connect to the proxy:
  export AWS_ACCESS_KEY_ID=ce940091...
  export AWS_SECRET_ACCESS_KEY=c0e32fb5...
  export AWS_CA_BUNDLE=/Users/gus/.tsh/keys/teleport.example.com/[email protected]/purple/awsconsole-localca.pem
  export HTTPS_PROXY=http://127.0.0.1:54430

# copy variables and re-paste into terminal
# (super annoying UX btw, we should support `tsh proxy aws --exec` or `--output aws-creds`
# so you can source the file easily)
gus@apollo:~ [fg: 1] %   export AWS_ACCESS_KEY_ID=ce940091...
  export AWS_SECRET_ACCESS_KEY=c0e32fb5...
  export AWS_CA_BUNDLE=/Users/gus/.tsh/keys/teleport.example.com/[email protected]/purple/awsconsole-localca.pem
  export HTTPS_PROXY=http://127.0.0.1:54430

# run command against ec2, works
gus@apollo:~ [fg: 1] % aws ec2 describe-instances --no-cli-pager
{
    "Reservations": [
        {
            "Groups": [],
            "Instances": [
                {
                    "AmiLaunchIndex": 0,
                    "ImageId": "ami-05912b6333beaa478",
                    "InstanceId": "i-08b794f12128dd450",
                    "InstanceType": "t3.medium",
<snip>

# wait a minute
gus@apollo:~ [fg: 1] % sleep 60

# re-run command, fails
gus@apollo:~ [fg: 1] % aws ec2 describe-instances

SSL validation failed for https://ec2.us-east-1.amazonaws.com/ [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1006)

# kill proxy
gus@apollo:~ [fg: 1] % fg
[1]  + 10617 running    tsh proxy aws
^C

# try to reauthenticate, fails due to HTTPS_PROXY being set
gus@apollo:~ % tsh app login awsconsole --aws-role PurpleAWSEC2FullAccess
ERROR: connection error: desc = "transport: Error while dialing: failed to dial: unable to establish proxy stream\n\trpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:54430: connect: connection refused\""

# angrily unset
gus@apollo:~ % unset HTTPS_PROXY

# reauthenticate again, provide MFA again
gus@apollo:~ % tsh app login awsconsole --aws-role PurpleAWSEC2FullAccess
MFA is required to access Application "awsconsole"
Tap any security key
Detected security key tap
Logged into AWS app "awsconsole".

Your IAM role:
  arn:aws:iam::<account>:role/PurpleAWSEC2FullAccess

Example AWS CLI command:
  tsh aws s3 ls

Or start a local proxy:
  tsh proxy aws --app awsconsole

# run proxy in background again
gus@apollo:~ % tsh proxy aws &
[1] 24772

gus@apollo:~ [fg: 1] % Started AWS proxy on http://127.0.0.1:54514.
To avoid port randomization, you can choose the listening port using the --port flag.

Use the following credentials and HTTPS proxy setting to connect to the proxy:
  export AWS_ACCESS_KEY_ID=17794ced...
  export AWS_SECRET_ACCESS_KEY=cc9d0564...
  export AWS_CA_BUNDLE=/Users/gus/.tsh/keys/teleport.example.com/[email protected]/purple/awsconsole-localca.pem
  export HTTPS_PROXY=http://127.0.0.1:54514

# copy/paste creds again, etc etc...

Bug details:

• Teleport version: 16.0.4

[Joerger]

This is working as expected - tsh app login creates MFA verified creds in the filesystem, so they shouldn't live longer than 1 minute. I added logic that allow the user to login and then tsh proxy app or tsh aws which opens a proxy under the hood, only prompting you for tap after 1 minute once it's needed. The second tap will result in longer lived certs stored in memory. Should I remove the first case and always promp for MFA immediately instea, ignoring previous tsh app logins?

Edit: relooked at the debug logs, is tsh proxy aws not prompting you for tap after a minute?

[webvictim]

Correct, I don't get a prompt - the operation just fails. Maybe because I'm not running it in the foreground?

I realised I'm using tsh 16.0.1 if that makes any difference. I don't think I saw anything in the changelog.

[Joerger]

Ah I should have added a changelog entry, I thought it was going to make it into v16.0.0 under the MFA for App access umbrella - #40985. It made it into v16.0.2 instead. Note you may run into this other issue before it's fixed.

[webvictim]

Got it - I'll update, retest, and then reopen the issue if anything new is broken. Thanks!