Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tsh proxy kube kube-cluster --cluster=leaf fails to reissue certs #41022

Closed
Joerger opened this issue Apr 29, 2024 · 0 comments · Fixed by #41046
Closed

tsh proxy kube kube-cluster --cluster=leaf fails to reissue certs #41022

Joerger opened this issue Apr 29, 2024 · 0 comments · Fixed by #41046
Assignees
@rosstimothy
Labels
bug c-q7j Internal Customer Reference tsh tsh - Teleport's command line tool for logging into nodes running Teleport.

Comments

@Joerger
Copy link
Contributor

Joerger commented Apr 29, 2024 •
edited by rosstimothy
Loading

When connecting to a kube cluster in a leaf cluster, tsh proxy kube fails to reissue certs successfully after the kube certs expire.

Fixing this behavior is especially important when max_session_ttl is low, which is common in per-session MFA setups.

Expected behavior:

tsh proxy kube kube-cluster --cluster=leaf should reissue certs whenever the certs expire.

Current behavior:

tsh proxy kube kube-cluster --cluster=leaf fails to reissue certs.

Debug logs:

2024-04-29T15:32:56-07:00 WARN [TSH]       Failed to reissue certificate for server 6d696e696b756265.leaf.example.com error:[
ERROR REPORT:
Original Error: *interceptors.RemoteError Kubernetes cluster "minikube" is not registered in this Teleport cluster; you can list registered Kubernetes clusters using 'tsh kube ls'
Stack Trace:
	github.com/gravitational/teleport/[email protected]/client/client.go:1128 github.com/gravitational/teleport/api/client.(*Client).GenerateUserCerts
	github.com/gravitational/teleport/lib/client/cluster_client.go:395 github.com/gravitational/teleport/lib/client.PerformMFACeremony
	github.com/gravitational/teleport/lib/client/client.go:561 github.com/gravitational/teleport/lib/client.(*ProxyClient).IssueUserCertsWithMFA
	github.com/gravitational/teleport/tool/tsh/common/kube_proxy.go:580 github.com/gravitational/teleport/tool/tsh/common.issueKubeCert
	github.com/gravitational/teleport/tool/tsh/common/kube_proxy.go:568 github.com/gravitational/teleport/tool/tsh/common.makeKubeLocalProxy.(*kubeLocalProxy).getCertReissuer.func1
	github.com/gravitational/teleport/lib/srv/alpnproxy/kube.go:243 github.com/gravitational/teleport/lib/srv/alpnproxy.(*KubeMiddleware).reissueCertIfExpired.func1
	runtime/asm_amd64.s:1650 runtime.goexit
User Message: Kubernetes cluster "minikube" is not registered in this Teleport cluster; you can list registered Kubernetes clusters using 'tsh kube ls'] alpnproxy/kube.go:181
2024-04-29T15:32:56-07:00 DEBU             Stopped forwarding request for "root.example.com:3080". alpnproxy/forward_proxy.go:374

Notes:

Teleport version: v15.2.4, v14.0.0

The cert checker local proxy middleware added in #40857 and expanded for http middleware in #40985 may be useful for fixing this issue.
@Joerger Joerger added the bug label Apr 29, 2024
@rosstimothy rosstimothy added the tsh tsh - Teleport's command line tool for logging into nodes running Teleport. label Apr 29, 2024
@rosstimothy rosstimothy mentioned this issue Apr 30, 2024
Merged
@rosstimothy rosstimothy self-assigned this Apr 30, 2024
rosstimothy added a commit that referenced this issue Apr 30, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube
793a240 
When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
@programmerq programmerq added the c-q7j Internal Customer Reference label Apr 30, 2024
rosstimothy added a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube
fea166c 
When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-merge-queue bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube (#…
d0e5a78 
…41046)

When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-actions bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube
0571550 
When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-actions bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube
4a1269b 
When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-actions bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube
3f10dba 
When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-merge-queue bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube (#…
20d5a69 
…41159)

When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-merge-queue bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube (#…
e489c0e 
…41157)

When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
github-merge-queue bot pushed a commit that referenced this issue May 2, 2024
@rosstimothy
Correctly reissue certificates for leaf resources in tsh proxy kube (#…
d9d2536 
…41158)

When renewing certificates the RouteToCluster was always being set
to the root cluster instead of the leaf cluster. This causes issues
with per session mfa because the root cluster can't find the target
kubernetes cluster which causes the renewal process to fail. Now
during renewal the RouteToCluster is copied from the active user
certificate if it existed.

Closes #41022.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rosstimothy rosstimothy

Labels
bug c-q7j Internal Customer Reference tsh tsh - Teleport's command line tool for logging into nodes running Teleport.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Correctly reissue certificates for leaf resources in tsh proxy kube gravitational/teleport
3 participants
@programmerq @Joerger @rosstimothy