diff --git a/docs/pages/enroll-resources/machine-id/troubleshooting.mdx b/docs/pages/enroll-resources/machine-id/troubleshooting.mdx
index c51225c3cb4b7..8945a7eddf3de 100644
--- a/docs/pages/enroll-resources/machine-id/troubleshooting.mdx
+++ b/docs/pages/enroll-resources/machine-id/troubleshooting.mdx
@@ -31,6 +31,13 @@ some additional context:
### Explanation
+
+This applies only to bots using the `token` join method, which makes use of
+one-time use shared secrets. Provider-specific join methods, such as GitHub,
+AWS IAM, etc will not be locked in this fashion unless another instance of the
+bot uses `token` joining.
+
+
Machine ID (with token-based joining) uses a certificate generation counter to
detect potentially stolen renewable certificates. Each time a bot fetches a new
renewable certificate, the Auth Service increments the counter, stores it on the
@@ -69,9 +76,11 @@ directories (usually `/opt/machine-id`) rather than the internal data directory
Once you have addressed the underlying cause, follow these steps to reset a
locked bot:
1. Remove the lock on the bot's user
- 1. Reset the bot's generation counter by deleting and re-creating the bot
+ 1. Reset the bot's generation counter by creating a new bot instance
-To remove the lock, first find and remove the lock targeting the bot user:
+To remove the lock, first find and remove the lock targeting the bot user. For
+this example, we'll assume the bot is named `example`, which will have an
+associated Teleport user named `bot-example`:
```code
$ tctl get locks
@@ -89,15 +98,16 @@ version: v2
$ tctl rm lock/5cee949f-5203-4f3b-9805-dac35d798a16
```
-Next, reset the generation counter by deleting and recreating the bot:
+Next, use `tctl bots instances add` to generate a new join token for the
+preexisting bot `example`:
```code
-$ tctl bots rm example
-
-$ tctl bots add example --roles=foo,bar
+$ tctl bots instances add example
```
-Finally, reconfigure the bot with the new token and restart it. It will detect
-the new token and automatically reset its internal data directory.
+Finally, reconfigure the local `tbot` instance with the new token and restart
+it. It will detect the new token and automatically reset its internal data
+directory. The bot will be issued a new bot instance UUID once connected, and
+the generation counter will be reset.
## `tbot` shows a "bad certificate error" at startup
@@ -123,8 +133,8 @@ fail."
Token-joined bots are unable to reauthenticate to the Teleport Auth Service once
their certificates have expired. Tokens in token-based joining (as opposed to
-AWS IAM joining) can only be used once, so when the bot's internal certificates
-expire, it will not be able to connect.
+AWS IAM and other join methods) can only be used once, so when the bot's
+internal certificates expire, it will not be able to connect.
When a bot's identity expires, certain parameters associated with the bot on the
Auth Service must be reset and a new joining token must be issued. The simplest
@@ -133,11 +143,10 @@ server-side data and issues a new joining token.
### Resolution
-Remove and recreate the bot, replacing the name and role list as desired:
+Use `tctl bots instances add` to create a new one-time use token for the bot:
```code
-$ tctl bots rm example
-$ tctl bots add example --roles=access
+$ tctl bots instances add example
```
Copy the resulting join token into the existing bot config—either the
diff --git a/docs/pages/reference/cli/tbot.mdx b/docs/pages/reference/cli/tbot.mdx
index 27b150d64bc45..4c474a7e96252 100644
--- a/docs/pages/reference/cli/tbot.mdx
+++ b/docs/pages/reference/cli/tbot.mdx
@@ -53,15 +53,20 @@ Additionally, be aware of the following limitations of `tbot db`:
## tbot init
-Initializes a certificate destination directory for access from a separate bot user. Allows for certificates to be written to disks other than a Machine ID client,
-configuring either file or POSIX ACL permissions.
+Initializes a certificate destination directory for access from a separate bot
+user. Allows for certificates to be written to disks other than a Machine ID
+client, configuring either file or POSIX ACL permissions.
+
+Note that most use cases should instead use tbot's runtime ACL management by
+specifying allowed reader users and groups in the
+[destination configuration](../machine-id/configuration.mdx#directory).
### Flags
| Flag | Description |
|---------------------|--------------------------------------------------------------------------------------------------------------------|
| `-d/--debug` | Enable verbose logging to stderr. |
-| `-c/--config` | Path to a Machine ID configuration file. |
+| `-c/--config` | Path to a Machine ID configuration file. |
| `--destination-dir` | Directory to write short-lived machine certificates to. |
| `--owner` | Defines the Linux `user:group` owner of `--destination-dir`. Defaults to the Linux user running `tbot` if unspecified. |
| `--bot-user` | Enables POSIX ACLs and defines the Linux user that can read/write short-lived certificates to `--destination-dir`. |
@@ -106,6 +111,10 @@ The `tbot proxy` command acts as a wrapper for `tsh proxy` to provide local prox
Note that `tsh` must be installed to make use of this command.
+Consider using one of the following dedicated tunnel modes where possible:
+- [`tbot start application-tunnel`](#tbot-start-application-tunnel)
+- [`tbot start database-tunnel`](#tbot-start-database-tunnel)
+
### Flags
| Flag | Description |
@@ -123,11 +132,11 @@ login step.
Additionally, the following should be noted:
- Certain CLI parameters, for example `--help`, may be captured by
-`tbot` even if intended to be passed to the wrapped `tsh`. A `--` argument can
-be used to ensure all following arguments are passed to `tsh` and ignored by
-`tbot`
+ `tbot` even if intended to be passed to the wrapped `tsh`. A `--` argument can
+ be used to ensure all following arguments are passed to `tsh` and ignored by
+ `tbot`
- If no configuration file is provided, `tbot` will apply a sample configuration based on provided CLI flags.
-For this reason, it is recommended that settings are explicitly applied to a configuration file in production.
+ For this reason, it is recommended that settings are explicitly applied to a configuration file in production.
### Examples
@@ -191,9 +200,100 @@ using database proxies.
| `--join-method` | Method to use to join the cluster. Can be `token` or `iam`. |
| `--oneshot` | If set, quit after the first renewal. |
+## tbot configure
+
+The `tbot configure` command is used to convert a `tbot start` CLI call into a
+YAML configuration file. It supports all the same subcommands and flags as
+[`tbot start`](#tbot-start), but prints an equivalent configuration instead of
+starting a bot.
+
+For example, consider this `tbot start identity` CLI call:
+```code
+$ tbot start identity \
+ --destination=./example \
+ --join-method=token \
+ --token=foo \
+ --proxy-server=teleport.example.com:443
+```
+
+To convert this CLI command into a configuration file, replace `tbot start` with
+`tbot configure`:
+```code
+$ tbot configure identity \
+ --destination=./example \
+ --join-method=token \
+ --token=foo \
+ --proxy-server=teleport.example.com:443
+```
+
+An equivalent YAML configuration will be printed, which can be written to a
+file. The client can then be started using this configuration file:
+
+```code
+$ tbot configure identity \
+ --destination=./example \
+ --join-method=token \
+ --token=foo \
+ --proxy-server=teleport.example.com:443 > tbot.yaml
+$ tbot start -c tbot.yaml
+```
+
+### Flags
+
+This subcommand supports one additional flag beyond its `tbot start` equivalent:
+
+| Flag | Description |
+|---------------|-------------|
+| `-o/--output` | If set, writes the generated configuration to the given file path instead of stdout. |
+
## tbot start
-Starts the Machine ID client `tbot`, fetching and writing certificates to disk at a set interval.
+The `tbot start` family of commands starts the Machine ID client in various
+modes, depending on the type of resources to be accessed:
+
+- [`tbot start legacy`](#tbot-start-legacy): Starts with a YAML configuration file or in legacy output mode
+- [`tbot start identity`](#tbot-start-identity): Starts with an identity output for SSH and Teleport API access
+- [`tbot start database`](#tbot-start-database): Starts with a database credential output
+- [`tbot start kubernetes`](#tbot-start-kubernetes): Starts with a Kubernetes output
+- [`tbot start application`](#tbot-start-application): Starts with an application TLS credential output
+- [`tbot start application-tunnel`](#tbot-start-application-tunnel): Starts a local application tunnel
+- [`tbot start database-tunnel`](#tbot-start-database-tunnel): Starts a local database tunnel
+- [`tbot start spiffe-svid`](#tbot-start-spiffe-svid): Starts a Workload ID SPIFFE SVID output
+
+If only `tbot start` is specified, `tbot start legacy` will be inferred by
+default; this is the correct mode for use with a YAML configuration file.
+
+### Common Start Flags
+
+These flags are available to all `tbot start` commands. Note that
+`tbot start legacy` supports slightly different options, so refer to its
+specific section for details when using a YAML config file or legacy output.
+
+| Flag | Description |
+|----------------------|-------------|
+| `-d/--debug` | Enable verbose logging to stderr. |
+| `--[no-]fips` | Whether to run tbot in FIPS compliance mode. This requires the FIPS `tbot` binary. |
+| `--log-format` | Controls the format of output logs. Can be `json` or `text`. Defaults to `text`. |
+| `-a/--auth-server` | Address of the Teleport Auth Service. Prefer using `--proxy-server` where possible. |
+| `--proxy-server` | Address of the Teleport Proxy Server. |
+| `--token` | A bot join token or path to file with token value, if attempting to onboard a new bot; used on first connect. |
+| `--ca-pin` | CA pin to validate the Teleport Auth Service; used on first connect. |
+| `--certificate-ttl` | TTL of short-lived machine certificates. |
+| `--renewal-interval` | Interval at which short-lived certificates are renewed; must be less than the certificate TTL. |
+| `--join-method` | Method to use to join the cluster. One of: `azure`, `circleci`, `gcp`, `github`, `gitlab`, `iam`, `kubernetes`, `spacelift`, `token`, `tpm`, `terraform_cloud` |
+| `--[no-]oneshot` | If set, quit after the first renewal. |
+| `--diag-addr` | If set and the bot is in debug mode, a diagnostics service will listen on specified address. |
+| `--storage` | A destination URI for tbot's internal storage, e.g. `file:///foo/bar`. See [Destination URIs](#destination-uris) for more info. |
+
+## tbot start legacy
+
+Starts the Machine ID client `tbot`, fetching and writing certificates to disk
+at a set interval. This command either starts from a configuration file if `-c`
+is specified, or starts with a default, legacy-compatible identity output.
+
+This is the default `tbot start` subcommand if no other command is specified.
+Unless using a configuration file, consider using `tbot start identity` or
+another dedicated mode instead.
### Flags
@@ -244,6 +344,189 @@ $ tbot start \
+## tbot start identity
+
+Starts the Machine ID client `tbot` with an identity output, fetching and
+writing certificates at a regular interval to the output specified with
+`--destination`.
+
+```code
+$ tbot start identity --destination=DESTINATION []
+```
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|----------------------|-------------|
+| `--destination` | A destination URI, such as file:///foo/bar. See [Destination URIs](#destination-uris) for more info. Required. |
+| `--reader-user` | An additional user name or UID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--reader-group` | An additional group name or GID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--cluster` | The name of a specific cluster for which to issue an identity if using a leaf cluster |
+
+### Examples
+
+To start a bot with a one-time-use join token:
+
+```code
+$ tbot start identity \
+ --proxy-server=example.teleport.sh:443 \
+ --join-type=token \
+ --token=TOKEN \
+ --destination=./tbot-user \
+ --storage=./tbot-data
+```
+
+## tbot start database
+
+Starts the Machine ID client `tbot` with a database output, fetching and writing
+database certificates at a regular interval to the output specified with
+`--destination`.
+
+```code
+$ tbot start database --destination=DESTINATION --service=SERVICE --username=USERNAME --database=DATABASE []
+```
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|----------------------|-------------|
+| `--destination` | A destination URI, such as file:///foo/bar. See [Destination URIs](#destination-uris) for more info. Required. |
+| `--reader-user` | An additional user name or UID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--reader-group` | An additional group name or GID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--format` | The database output format if necessary |
+| `--service` | The service name of the database as it appears in Teleport and `tsh db ls`. Required. |
+| `--username` | The database user name. The bot user must have permission to connect as this user. Required. |
+| `--database` | The name of the database available in the requested service. Required. |
+
+## tbot start kubernetes
+
+Starts the Machine ID client `tbot` with a Kubernetes output, fetching and
+writing Kubernetes credentials and a `kubeconfig.yaml` at a regular interval to
+the output specified with `--destination`.
+
+```code
+$ tbot start kubernetes --destination=DESTINATION --kubernetes-cluster=KUBERNETES-CLUSTER []
+```
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|------------------------|-------------|
+| `--destination` | A destination URI, such as file:///foo/bar. See [Destination URIs](#destination-uris) for more info. Required. |
+| `--reader-user` | An additional user name or UID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--reader-group` | An additional group name or GID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--kubernetes-cluster` | The name of the Kubernetes cluster in Teleport for which to fetch credentials |
+
+## tbot start application
+
+Starts the Machine ID client `tbot` with an application output, fetching and
+writing application TLS credentials at a regular interval to the output
+specified with `--destination`.
+
+```code
+$ tbot start application --destination=DESTINATION --app=APP []
+```
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|----------------------|-------------|
+| `--destination` | A destination URI, such as file:///foo/bar. See [Destination URIs](#destination-uris) for more info. Required. |
+| `--reader-user` | An additional user name or UID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--reader-group` | An additional group name or GID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--app` | The name of the app in Teleport |
+
+## tbot start application-tunnel
+
+Starts the Machine ID client with a local tunnel to a particular application.
+This tunnel will run continuously and automatically refresh its certificates.
+
+Note that this tunnel will be unencrypted. Be wary of the selected listen
+address, and prefer to use `localhost` or an equivalent loopback interface
+address when possible. Additionally, note that all users on the local system
+will be able to access this socket.
+
+If you wish to tunnel multiple apps from one bot instance, use
+`tbot configure application-tunnel ...` to generate a configuration file and
+repeat the generated block under `services:` as desired.
+
+This tunneling method is preferred over the legacy [`tbot proxy app`](#tbot-proxy).
+
+```code
+$ tbot start application-tunnel --listen=LISTEN --app=APP []
+```
+
+### Flags
+
+| Flag | Description |
+|------------|-------------|
+| `--listen` | A socket URI to listen on, e.g. `tcp://localhost:1234`. Required. |
+| `--app` | The name of the app in Teleport |
+
+## tbot start database-tunnel
+
+Starts the Machine ID client with a local tunnel to a particular database.
+This tunnel will run continuously and automatically refresh its certificates.
+
+Note that this tunnel will be unencrypted. Be wary of the selected listen
+address, and prefer to use `localhost` or an equivalent loopback interface
+address when possible. Additionally, note that all users on the local system
+will be able to access this socket.
+
+If you wish to tunnel multiple databases from one bot instance, use
+`tbot configure database-tunnel ...` to generate a configuration file and repeat
+the generated block under `services:` as desired.
+
+This tunneling method is preferred over the legacy
+[`tbot proxy db`](#tbot-proxy), and is roughly equivalent to
+`tbot proxy db --tunnel`.
+
+```code
+$ tbot start database-tunnel --listen=LISTEN --service=SERVICE --username=USERNAME --database=DATABASE []
+```
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|--------------|-------------|
+| `--listen` | A socket URI to listen on, e.g. `tcp://localhost:1234`. Required. |
+| `--service` | The service name of the database as it appears in Teleport and `tsh db ls`. Required. |
+| `--username` | The database user name. The bot user must have permission to connect as this user. Required. |
+| `--database` | The name of the database available in the requested service. Required. |
+
+## tbot start spiffe-svid
+
+### Flags
+
+In addition to the [common `tbot start` flags](#common-start-flags), this
+command supports these additional flags:
+
+| Flag | Description |
+|----------------------|-------------|
+| `--destination` | A destination URI, such as file:///foo/bar. See [Destination URIs](#destination-uris) for more info. Required. |
+| `--reader-user` | An additional user name or UID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--reader-group` | An additional group name or GID that should be allowed by ACLs to read this destination. Only valid for file destinations on Linux. |
+| `--[no-]include-federated-trust-bundles` | If set, include federated trust bundles in the output |
+| `--svid-path` | A SPIFFE ID to request, prefixed with '/'. Required. |
+| `--svid-hint` | An optional hint for consumers of the SVID to aid in identification |
+| `--dns-san` | A DNS name that should be included in the SVID. Repeatable. |
+| `--ip-san` | An IP address that should be included in the SVID. Repeatable. |
+
## tbot install systemd
Generates and installs a systemd unit file for a specific tbot configuration.
@@ -266,5 +549,23 @@ Generates and installs a systemd unit file for a specific tbot configuration.
```code
$ tbot install systemd \
--config=/etc/tbot.yaml \
- ---write
-```
\ No newline at end of file
+ --write
+```
+
+## Destination URIs
+
+Many `tbot start` subcommands accept destination URIs via the `--storage` and
+`--destination` flags.
+
+| Protocol | Description |
+|------------------------|-------------|
+| `file://` | A local directory destination, such as `file:///foo/bar/` |
+| `memory://` | An in-memory destination. Useful for internal bot storage if no persistence is required. |
+| `kubernetes-secret://` | A Kubernetes secret destination, such as `kubernetes-secret:///my-secret` |
+
+Plain file paths are also be accepted with no `file://` prefix; these will be
+treated as directory outputs.
+
+Note that `tbot start legacy` only supports directory output destinations via
+the `--destination-dir` flag, though it does support URIs for `--storage`. All
+other `tbot start` subcommands accept these URIs where relevant.
diff --git a/docs/pages/reference/cli/tctl.mdx b/docs/pages/reference/cli/tctl.mdx
index d79863d0f5d3c..e7f65fd3017c8 100644
--- a/docs/pages/reference/cli/tctl.mdx
+++ b/docs/pages/reference/cli/tctl.mdx
@@ -426,6 +426,95 @@ $ tctl bots add example --roles=access --logins=root
These flags are available for all commands: `--debug, --config`. Run
`tctl help ` or see the [Global Flags section](#tctl-global-flags).
+## tctl bots instances add
+
+Onboard a new instance of an existing bot with a one-time use join token:
+
+```code
+$ tctl bots instances add [flags]
+```
+
+Note that a new instance will not be generated immediately; the bot will appear
+in `tctl bots instances list` as soon as it joins the cluster and is allocated a
+UUID.
+
+### Arguments
+
+- `` - the name of the existing bot for which a new join token should be
+ generated
+
+### Flags
+
+| Name | Default Value(s) | Allowed Value(s) | Description |
+| - | - | - | - |
+| `--token` | none | String name of an existing join token. | Optional. Specifies an existing join token to be used rather than creating one as part of the Bot creation. |
+| `--format` | `text` | `text`, `json` | If set to `json`, return new bot information as a machine-readable JSON string. |
+
+### Examples
+
+This generates a new one-time use join token for the bot named "example",
+without modifying roles or other configuration:
+
+```code
+$ tctl bots instances add example
+```
+
+### Global flags
+
+These flags are available for all commands: `--debug, --config`. Run
+`tctl help ` or see the [Global Flags section](#tctl-global-flags).
+
+## tctl bots instances list
+
+List all current instances of a Machine ID bot:
+
+```code
+$ tctl bots instances list []
+```
+
+### Arguments
+
+- `[]` - an optional bot name. If provided, filters the result to show
+ only instances for the named bot. Otherwise, shows all instances for all bots.
+
+### Examples
+
+This shows all known instances for the bot named "example":
+
+```code
+$ tctl bots instance list example
+```
+
+### Global flags
+
+These flags are available for all commands: `--debug, --config`. Run
+`tctl help ` or see the [Global Flags section](#tctl-global-flags).
+
+## tctl bots instances show
+
+Shows details about a particular instance of a bot, including join metadata and
+current status:
+
+```code
+$ tctl bots instances show /
+```
+
+### Arguments
+
+- `` - the name of the existing bot
+- `` - the UUID of the existing bot instance, as shown in `tctl bots instance list`
+
+### Examples
+
+```code
+$ tctl bots instances show example/28e605c9-2fe1-423f-afe1-5122335e1c75
+```
+
+### Global flags
+
+These flags are available for all commands: `--debug, --config`. Run
+`tctl help ` or see the [Global Flags section](#tctl-global-flags).
+
## tctl bots ls
Lists all Machine ID Bots:
diff --git a/docs/pages/reference/machine-id/configuration.mdx b/docs/pages/reference/machine-id/configuration.mdx
index e571de8dc5d19..5974df5f81038 100644
--- a/docs/pages/reference/machine-id/configuration.mdx
+++ b/docs/pages/reference/machine-id/configuration.mdx
@@ -8,11 +8,10 @@ This reference documents the various options that can be configured in the `tbot
configuration file. This configuration file offers more control than
configuring `tbot` using CLI parameters alone.
-To configure `tbot` to use a configuration file, specify the path with the `-c`
-flag:
+To run `tbot` with a configuration file, specify the path with the `-c` flag:
```code
-$ tbot -c ./tbot.yaml
+$ tbot start -c ./tbot.yaml
```
In this reference, the term **artifact** refers an item that `tbot` writes to a
@@ -918,6 +917,7 @@ path: /opt/machine-id
# if unsupported.
# * insecure: Quietly allow symlinks in paths.
symlinks: try-secure
+
# acls configures whether Linux Access Control List (ACL) setup should occur for
# this destination.
# Requires Linux with a file system that supports ACLs.
@@ -928,6 +928,21 @@ symlinks: try-secure
# * required: Always use ACLs, produce a hard error at runtime if ACLs
# are invalid.
acls: try
+
+# readers is a list of users and groups that will be allowed by ACL to access
+# this directory output. The `acls` parameter must be either `try` or
+# `required`. File ACLs will be monitored and corrected at runtime to ensure
+# they match this configuration.
+# Individual entries may either specify `user` or `group`, but not both. `user`
+# accepts an existing named user or a UID, and `group` accepts an existing named
+# group or GID. UIDs and GIDs do not necessarily need to exist on the local
+# system.
+# An empty list of readers disables runtime ACL management.
+readers:
+- user: teleport
+- user: 123
+- group: teleport
+- group: 456
```
### `memory`