Discovers services running in a Kubernetes cluster by
querying the Kubernetes API server. This observer is designed to only
discover pod endpoints exposed on the same node that the agent is running,
so that the monitoring of services does not generate cross-node traffic. To
know which node the agent is running on, you should set an environment
variable called MY_NODE_NAME using the downward API spec.nodeName value
in the pod spec. Our provided K8s DaemonSet resource does this already and
provides an example.
Note that this observer discovers exposed ports on pod containers, not K8s Endpoint resources, so don't let the terminology of agent "endpoints" confuse you.
Observer Type: k8s-api
| Config option | Required | Type | Description |
|---|---|---|---|
namespace |
no | string |
If specified, only pods within the given namespace on the same node as the agent will be discovered. If blank, all pods on the same node as the agent will be discovered. |
kubernetesAPI |
no | object (see below) |
Configuration for the K8s API client |
additionalPortAnnotations |
no | list of strings |
A list of annotation names that should be used to infer additional ports to be discovered on a particular pod. The pod's annotation value should be a port number. This is useful for annotations like prometheus.io/port: 9230. If you don't already have preexisting annotations like this, we recommend using the SignalFx-specific annotations. |
The nested kubernetesAPI config object has the following fields:
| Config option | Required | Type | Description |
|---|---|---|---|
authType |
no | string |
How to authenticate to the K8s API server. This can be one of none (for no auth), tls (to use manually specified TLS client certs, not recommended), serviceAccount (to use the standard service account token provided to the agent pod), or kubeConfig to use credentials from ~/.kube/config. (default: serviceAccount) |
skipVerify |
no | bool |
Whether to skip verifying the TLS cert from the API server. Almost never needed. (default: false) |
clientCertPath |
no | string |
The path to the TLS client cert on the pod's filesystem, if using tls auth. |
clientKeyPath |
no | string |
The path to the TLS client key on the pod's filesystem, if using tls auth. |
caCertPath |
no | string |
Path to a CA certificate to use when verifying the API server's TLS cert. Generally this is provided by K8s alongside the service account token, which will be picked up automatically, so this should rarely be necessary to specify. |
The following fields are available on endpoints generated by this observer and can be used in discovery rules.
| Name | Type | Description |
|---|---|---|
container_name |
string |
The first and primary name of the container as it is known to the container runtime (e.g. Docker). |
ip_address |
string |
The IP address of the endpoint if the host is in the from of an IPv4 address |
network_port |
string |
An alias for port |
private_port |
string |
The port that the service endpoint runs on inside the container |
public_port |
string |
The port exposed outside the container |
alternate_port |
integer |
Used for services that are accessed through some kind of NAT redirection as Docker does. This could be either the public port or the private one. |
container_command |
string |
The command used when running the container exposing the endpoint |
container_id |
string |
The ID of the container exposing the endpoint |
container_image |
string |
The image name of the container exposing the endpoint |
container_labels |
map of string |
A map that contains container label key/value pairs. You can use the Contains and Get helper functions in discovery rules to make use of this. See Endpoint Discovery. |
container_names |
list of string |
A list of container names of the container exposing the endpoint |
container_state |
string |
The container state, will usually be "running" since otherwise the container wouldn't have a port exposed to be discovered. |
discovered_by |
string |
The observer that discovered this endpoint |
host |
string |
The hostname/IP address of the endpoint |
id |
string |
|
name |
string |
A observer assigned name of the endpoint |
orchestrator |
integer |
|
port |
integer |
The TCP/UDP port number of the endpoint |
port_labels |
map of string |
A map of labels on the container port. You can use the Contains and Get helper functions in discovery rules to make use of this. See Endpoint Discovery. |
port_type |
string |
TCP or UDP |
These dimensions are added to all metrics that are emitted for this service endpoint. These variables are also available to use as variables in discovery rules.
| Name | Description |
|---|---|
container_image |
The image name (including tags) of the running container |
container_name |
The primary name of the running container -- Docker containers can have multiple names but this will be the first name, if any. |
container_spec_name |
The short name of the container in the pod spec, NOT the running container's name in the Docker engine |
kubernetes_namespace |
The namespace that the discovered service endpoint is running in. |
kubernetes_pod_name |
The name of the running pod that is exposing the discovered endpoint |
kubernetes_pod_uid |
The UID of the pod that is exposing the discovered endpoint |