Added new OpenSSH break glass admin management operations guide #49804
Conversation
Signed-off-by: Alen Haric <[email protected]>
github-actions
bot
requested review from
mmcallister,
r0mant,
xinding33 and
zmb3
deusxanima
added
the
no-changelog
|
🤖 Vercel preview here: https://docs-5h5xxnof7-goteleport.vercel.app/docs |
| sudo visudo | ||
| ``` | ||
|
|
||
| 3. Grant limited sudo permissions: |
There was a problem hiding this comment.
There is no way to grant limited permissions while allowing for emergency system maintenance; case in point, we suggest systemctl which, as root, can just spawn any command with root privileges as a unit (among many other ways) and we suggest vim, which can just edit /etc/sudoers to begin with.
There was a problem hiding this comment.
Hmm...great points. Would it make more sense then to just remove this section entirely?
There was a problem hiding this comment.
Honestly, sometimes people want a little security theatre - restricted shells are a very common request.
With that said, I think for break glass access having command restrictions may inhibit people from fixing issues and it's best to just allow root permissions through sudo. Any use of the break glass access should always be audited via sshd access logs anyway.
There was a problem hiding this comment.
great feedback. I'll clean it up and put up a new version with the recommended changes/feedback
| Run the following after logging into Teleport via `tsh login` as the impersonating user: | ||
|
|
||
| ```bash | ||
| tctl auth sign --ttl 24h --user=breakglass --out=/safe/path/breakglass --format openssh |
There was a problem hiding this comment.
This user certificate is (or, at least, I really hope is) issued by the user CA, which is the signer that signs all the user certs that are given out to users, but above we configured opensshd to trust the openssh CA, which signs certificates that are only ever in possession of the proxy - a regular user won't be able to obtain such a certificate.
There was a problem hiding this comment.
Interesting. From what I can see, it looks like the CA that's exported with the proxy via the https://proxy.example.com/webapi/auth/export?type=openssh endpoint, is of type "user" instead of OpenSSH according to the extension appended at the end:
e.g.:
cert-authority ssh-rsa <my_ca> clustername=alen.teleport.sh&type=user
I can confirm that passing type=user instead during the export produces a different CA, but if the former is not expected to work with user certs generated via tctl auth sign ... --format openssh, there may be another problem altogether as I can get it to work reliably right now (unless again I'm turned around on this and misunderstanding, which very likely could be the case)
| Run the following after logging into Teleport via `tsh login` as the impersonating user: | ||
|
|
||
| ```bash | ||
| tctl auth sign --ttl 24h --user=breakglass --out=/safe/path/breakglass --format openssh |
There was a problem hiding this comment.
Is the cluster admin supposed to renew this break glass cert every day? If not, how are you supposed to get a certificate signed by the control plane if the control plane is down?
There was a problem hiding this comment.
That was my recommendation a bit further down in this section:
Rotate both keys and certificates regularly, making sure to refresh both before the expiration TTL
There was a problem hiding this comment.
We probably need to make this a lot more obvious so people don't inadvertently get locked out. Maybe describe how to run the command as a cron job or systemd timer which will write a certificate every 12 hours so it can be picked up?
It would also be good to describe how to do this end to end with a machine ID job - that can be a task for version 2 of the guide though.
| 2. Create an authorized_principals file for the `breakglass` user | ||
|
|
||
| ```bash | ||
| echo "breakglass" | sudo tee /etc/ssh/authorized_principals/breakglass |
There was a problem hiding this comment.
Instead of this whole machinery, can't we just create a breakglass user and make it trust the appropriate CA by putting it in ~breakglass/.ssh/authorized_keys with the appropriate principals restriction?
There was a problem hiding this comment.
We could even do that for the root user, and make it require a principal of root-breakglass or something that won't accidentally be assigned as a login name to regular Teleport users.
There was a problem hiding this comment.
Instead of this whole machinery, can't we just create a breakglass user and make it trust the appropriate CA by putting it in ~breakglass/.ssh/authorized_keys with the appropriate principals restriction?
I actually had this in the original setup but after some discussion with a few other folks went with the authorized_principals approach instead for the PR. Happy to switch it back if you think it simplifies things, and I like your recommendation of also doing the same thing for the root user so if makes sense to switch back I can tackle both at once in the updated commit.
|
Hi @deusxanima, would this PR address #3760? It's one of our oldest docs issues, so it'd be great to see it cleared! |
Added guide with instructions on how to configure OpenSSH break glass access with Teleport CA signed certificates for out-of-band access if Teleport server resources or cluster servers become unavailable and/or unresponsive.