remote-config.md

Remote Configuration

The agent always requires a main config file on a local filesystem, but values within that config can pull from other sources. These sources include other files on the filesystem or KV stores such as Zookeeper, Etcd, and Consul. Additional stores can be easily added.

A remote config value looks like this in the main config file:

signalFxAccessToken: {"#from": "/etc/signalfx/token"}

The remote config specification is a YAML map object that includes the #from key. The value of that key is the path from which to get the config value.

The value of the #from key has the form <source>:<path>. If it is only a path with no <source>:, it is assumed to be a file path on the local filesystem. All non-filesystem sources must be configured in the sourceConfig section of the main config file.

If the source is a reference to a single path, that value is deserialized by YAML and inserted into the config as if it had been literally in the config as such. The replacement is done in such a way that you don't need to worry about matching indentation of remote values.

Configuration of Remote Configuration

The sources for remote configuration can be configured via the configSources option in the main agent config file. A full list of config sources and their options are also documented in the config reference.

Simple paths

The most basic way to reference a value is to use a single, non-globbed path. The content stored at that path can be simple scalar values such as strings or integers, or YAML collections such as sequences or maps. The only requirement is that they deserialize from YAML properly. Note that JSON is a subset of YAML, so any valid JSON can also be used.

Nested Values (Vault only)

Only supported by Vault remote config.

The Vault remote config source supports reaching into secret data that is of a map type and pulling out a specific value. This is necessary in Vault because of how secrets are structured.

Accessing nested data uses a syntax similar to many programming languages for accessing a "map-like" object: square brackets. E.g. to access the password field of a Vault secret, you would use the following:

{"#from": "vault:secret/my-database[password]"}

You can also reach into nested map values by separating the keys by .. For example, when using the KV v2 secret engine in Vault, all data will be nested under a separate data map within the secret. You would access the same password field in that by doing:

{"#from": "vault:secret/data/my-database[data.password]"}

If the given key doesn't exist in the secret, the value will resolve to null and the agent config will fail to load (unless the option has optional: true on it, see below).

Unless debug logging is enabled, the secret values will never be logged.

Globbed paths

Not supported by Vault remote config.

Some config sources support globbing in the source path. If there is a glob in the source path, the YAML content of the matching paths will be read and deserialized. All of the values must be YAML collections of the same type (i.e. either all a sequence or all a map), or else an error is raised. All of those collections will be merged together and treated as one collection.

Flattening

You can flatten both sequences and maps into the parent of the item where the remote value is specified. For example:

monitors:
 - type: collectd/mysql
   username: signalfx
   databases:
    - name: admin
    - {"#from": "zk:/signalfx-agent/mysql/databases/*", flatten: true}

Given the following znodes name and values in ZooKeeper:

• /signalfx-agent/mysql/databases/app1: [{name: my-db1}, {name: my-db2}]
• /signalfx-agent/mysql/databases/app2: [{name: their-db1}, {name: their-db2}]

The final resolved config would look like this:

monitors:
 - type: collectd/mysql
   username: signalfx
   databases:
    - name: admin
    - name: my-db1
    - name: my-db2
    - name: their-db1
    - name: their-db2

Optional paths

You may want to allow for globbed paths that don't actually match anything. This is very useful for specifying a directory of extra monitor configurations that may be empty, such as the following:

signalFxAccessToken: abcd
monitors:
 - {"#from": "/etc/signalfx/conf2/*.yaml", flatten: true, optional: true}
 - type: collectd/cpu
 - type: collectd/cpufreq
 - type: collectd/df

The key here is the optional: true value, which makes it accept globs that don't match anything. optional defaults to false so it must be explicitly stated that you are ok with no matches.

optional also works in scalar contexts as well, assuming that the config value is not required by the agent.

Raw Values

If you have values in files/KV stores that you don't want interpreted as YAML, but rather as plain strings, you can add the raw: true option to the remote value specification. Everything else acts as it would otherwise.

Environment Variables

The config file also supports environment variable interpolation with the ${VARNAME} syntax (the curly brackets are required). Environment variables are interpolated after all of the remote config values are interpolated, which means that remote config values can contain references to envvars, if so desired. Envvars cannot, however, contain remote config values.

For more advanced environment variable interpolation you can use the remote config value syntax with the env type. For example:

signalFxAccessToken: {"#from": "env:SIGNALFX_ACCESS_TOKEN"}

will interpolate the given environment variable to that place in the config file. The advantage of this format over the plain ${VARNAME} syntax is that you can use all of the advanced features of remote config, such as flattening and defaults, which are not possible with the ${} syntax.

The env var remote config source does not pick up changes to envvars that happen after the initial source resolution.

Other

If you need more sophisticated interpolation of config values from KV stores, we recommend using a third-party templating solution such as confd, or writing your own custom script.