Splunk Configuration

Introduction

Splunk configuration is done on an inventory group basis.

For each inventory group there is a group_vars directory that accepts variable files. The variable files have a one-to-one relation to Splunk configuration files (.conf).

Let's say, an inputs.conf configuration has to be deployed to all search peers, a file named group_vars/searchpeer/inputs.conf needs to be created.

Note: It is possible to overwrite configurations on a single host basis by using host_vars. Precedence of namespaced variable differs in Ansible by default. For further information, see Ansible Documentation.

Variable files reflect the configurations that will be written to the Splunk conf files. There is a different in syntax, as Splunk keeps configuration in INI-file format, while Ansible stores configurations in YAML format. Also, for advanced functionality, variables are often written in a different format.

Note: Not all Splunk INI configuration items are implemented. If an important item is missing, please open an Issue

Configuration Files (Specifications)

Specification Files

All specification files are located under README/spec

distsearch_conf.spec:

############################################
#
# Possible values for conf/distsearch role
#
# Follows Splunk distsearch.conf.spec closely
#
############################################

splunk_distsearch_conf:
  distributedSearch:
    disabled: [True | False]
    * Defaults to false
  server:
    - "{{ groups['<groupname1>'] }}"
    - "{{ groups['<groupname2>'] }}"
    - ...
    - "{{ groups['<groupnameN>'] }}"
   * List of inventory groups used for distributed search

inputs_conf.spec:

############################################
#
# Possible values for conf/inputs role
# 
# Follows Splunk inputs.conf.spec closely
#
############################################

splunk_inputs_conf:
  splunktcp: 
    port: <port>

  splunktcp_ssl: 
    port: <port>

  SSL:
    rootCA: <filepath>
    * Certificate authority list
    * Autogenerated file under $SPLUNK_HOME/etc/auth/cacert.pem

    serverCert: <filepath>
    * Full path to the server certificate.
    * Autogenerated file under $SPLUNK_HOME/etc/auth/server.pem

    password: <string>
    * Encrypted password

outputs_conf.spec:

############################################
#
# Possible values for conf/outputs role
#
# Follows Splunk outputs.conf.spec closely
#
############################################

splunk_outputs_conf:
  tcpout:
    defaultGroup: <target_group>, <target_group>, ...
    * The default group
    target_group:
      [target_group1:]
      * List of target groups

        useACK: [true|false]
        * Defaults to false
        server:
          - ["<server>:<port>"]
          - ["<server>:<port>"]
          - ...
          * List of servers to connect to

        sslPassword: <password>
        * Encrypted sslPassword
        * No default value

        sslCertPath: <path>
        * There is no default value.
        * Autogenerated file under $SPLUNK_HOME/etc/auth/server.pem

        sslRootCAPath: <path>
        * There is no default value.
        * Autogenerated file under $SPLUNK_HOME/etc/auth/ca.pem

        sslVerifyServerCert: [true|false]
        * Defaults to false.

        indexerDiscovery: <name>
        * Instructs the forwarder to fetch the list of indexers from the master node specified in the corresponding [indexer_discovery:<name>] stanza.

      [target_groupN:]
      ...

  indexer_discovery:
    * Indexer discovery settings

    [indexerDiscovery_target1:]
    * indexerDiscpvery_target list. Use name from tcpout stanza.

      pass4SymmKey: <password>
      * Encrypted Password

      master_uri: "<uri>"
      * Cluster Master URI
      * No default value

server_conf.spec

############################################
#
# Possible values for conf/server role
# 
# Follows Splunk server.conf.spec closely
#
############################################

splunk_server_conf:
  general:
    pass4Symmkey: <password>
    * Encrypted password

    site: [dynamic|<site>]
    * The site where the system is located. Dynamic will use site attribute from inventory

    trustedIP: <IP address>
    * Trusted IP for SSO

  diskUsage:
    minFreeSpace: <num>
    * Specified in megabytes.
    * The default setting is 5000 (approx 5GB)

  sslConfig:
    sslKeysfilePassword: <password>
    * Encrypted password 

  license:
    master_uri: [dynamic|<uri>]
    * Dynamic will use inventory to detect

  clustering:
    mode: [master|slave|searchhead|disabled]
    * Defaults to disabled

    master_uri: [<uri> | clustermaster:stanzaName1, clustermaster:stanzaName2]
    * URI of the cluster master that this slave or searchhead should connect to.
    
    pass4SymmKey: <password>
    * Encrypted password
 
    multisite: [true|false]
    * Defaults to false

    replication_factor: <positive integer>
    * Defaults to 3

    site_replication_factor: <comma-separated string>
    * Defaults to origin:2,total:3
    * Note: no spaces allowed between comma separated values

    search_factor:  <positive integer>
    * Defaults to 2

    available_sites:
      - [site1]
      - [site2]
     * List of available sites
     * Defaults to an empty string. So if multisite is turned on this needs to be explicitly set

     cluster_label: [dynamic|<string>]
     * Defaults to an empty string.
     * Use dynamic to set automatically

  replication_port:
    port: <port>
    * Replication port

  replication_port_ssl:
    port: <port>

    rootCA: <filepath>
    * Certificate authority list
    * Autogenerated file under $SPLUNK_HOME/etc/auth/cacert.pem

    serverCert: <filepath>
    * Full path to the server certificate.
    * Autogenerated file under $SPLUNK_HOME/etc/auth/server.pem

    password: <string>
    * Encrypted password
  
  shclustering:
    mgmt_uri: [mgmt-URI | dynamic]
    * The management uri is used to identify the cluster members own address to
      itself.
    * Use dynamic to set own adress automatically
   
    id: <GUID>
    * Unique identifier for this cluster as a whole, shared across all cluster
    members.
    * Create one, e.g. using python: $ python -c "import uuid; print str(uuid.uuid4()).upper()" 

    conf_deploy_fetch_url: [ <URL> | dynamic ]
    * Specifies the location of the deployer from which members fetch the
      configuration bundle.
    * This value must be set to a <URL> or dynamic in order for the configuration bundle to
      be fetched.
    * Set to dynamic to automatically set value
    * Defaults to empty.

    election: [True | False]
    * This is used to classify a cluster as static or dynamic (RAFT based). 
    * election = false means static captain, which is used for DR situation.
    * election = true means dynamic captain election enabled through RAFT protocol

    pass4SymmKey: <password>
    * Secret shared among the members in the search head cluster to prevent any
      arbitrary instance from connecting to the cluster.
    * All members must use the same value.
    * If set in the [shclustering] stanza, it takes precedence over any setting
      in the [general] stanza.

    replication_factor: <positive integer>
    * Determines how many copies of search artifacts are created in the cluster.
    * This must be set to the same value on all members.
    * Defaults to 3.

    shcluster_label = <string>
    * This specifies the label of the search head cluster

web_conf_spec:

############################################
#
# Possible values for conf/web role
# 
# Follows Splunk web.conf.spec closely
#
############################################

splunk_web_conf:
  settings:
    enableSplunkWebSSL: [True | False]
    * Defaults to True

    httpport: <port> 
    * Defaults to 8000

    startwebserver: [True | False]
    * Defaults to True

    privKeyPath: <path>
    * Relative paths are interpreted as relative to $SPLUNK_HOME
    * Defaults to etc/auth/splunkweb/privkey.pem

    caCertPath:  <path>
    * Relative paths are interpreted as relative to $SPLUNK_HOME
    * Default to etc/auth/splunkweb/cert.pem

    updateCheckerBaseURL: [http://quickdraw.Splunk.com/js/|0]
    * Defaults to http://quickdraw.Splunk.com/js/

Configuration Files (Samples)

Sample distsearch.conf:

---

#####################################################################################################################
# Configurations for Splunk distsearch.conf
####################################################################################################################

splunk_distsearch_conf:
  distributedSearch:
    servers: 
       - "{{ groups['searchpeer'] }}"
       - "{{ groups['peernode'] }}"
       - "{{ groups['licensemaster'] }}"
       - "{{ groups['masternode'] }}"
       - "{{ groups['deploymentserver'] }}"
       - "{{ groups['deployer'] }}"
       - "{{ groups['shcmember'] }}"
       - "{{ groups['heavyforwarder'] }}"

Sample inputs.conf:

---

#####################################################################################################################
# Configurations for Splunk inputs.conf
#####################################################################################################################

splunk_inputs_conf:
  splunktcp_ssl: 
    port: 9997
  SSL:
    rootCA: $SPLUNK_HOME/etc/auth/cacert.pem
    serverCert: $SPLUNK_HOME/etc/auth/server.pem
    password: $1$2lO4zAA+GjIL

Sample outputs.conf:

---

#####################################################################################################################
# Configurations for Splunk outputs.conf
#####################################################################################################################

splunk_outputs_conf:
  tcpout:
    defaultGroup: production
    target_group: 
      production:
        useACK: true
        indexerDiscovery: production
        sslPassword: password
        sslCertPath: $SPLUNK_HOME/etc/auth/server.pem
        sslRootCAPath: $SPLUNK_HOME/etc/auth/ca.pem
  indexer_discovery:
    production:
      pass4SymmKey: changeme
      master_uri: dynamic

Sample server.conf

---

#####################################################################################################################
# Configurations for Splunk server.conf
####################################################################################################################

splunk_server_conf:
  diskUsage:
    minFreeSpace: 2000
  license:
    master_uri: dynamic
  clustering:
    mode: searchhead
    master_uri: dynamic
  shclustering:
    shcluster_label: production
    mgmt_uri: dynamic
    conf_deploy_fetch_url: dynamic
    id: 6D1F6C76-3370-40BA-98F3-2C16AEEF24F1
  replication_port_ssl:
    port: 9888
    rootCA: $SPLUNK_HOME/etc/auth/cacert.pem
    serverCert: $SPLUNK_HOME/etc/auth/server.pem
    password: $1$2lO4zAA+GjI

Sample web.conf:

---

#####################################################################################################################
# Configurations for Splunk web.conf
#####################################################################################################################

splunk_web_conf:
  settings:
    enableSplunkWebSSL: 1
    updateCheckerBaseURL: 0

More Samples

More samples can be found under README/templates/group_vars