Splunk TLS

A practice guide to enabling TLS in your Splunk Environment

Within documentation and education, Splunk often uses the terms SSL and TLS interchangeably. This is a practical walkthrough for enabling a line of secure communication between your indexer(s) and forwarder(s), and on the web front end of your search head(s).

Before we get into the processes within Splunk, we are going to set up a quick Certificate Authority. This works well when you are running a local instance, lab, or development environment.

Setting Up the Certificate Authority

  1. Create a directory to work within:
    mkdir /opt/internalCA
  2. Generate a root key and security it with a password:
    openssl genrsa -des3 -out /opt/internalCA/rootCA.key 2048
  3. Store the password as necessary
  4. Now use openssl to request a new certificate, signed with our rootCA.key
    openssl req -x509 -new -nodes -key /opt/internalCA/rootCA.key -sha256 -days 365 -out /opt/internalCA/rootCA.pem
  5. Take the rootCA.pem and distribute it to the workstation you want. For the purposes of this lab, install it into the browser you access Splunk with.

Examples: Firefox (Preferences → Certificate Authorities → Import)

It’ll be listed under Internet Widgits

There are two specific commands that we’ll be using from our new certificate authority.

Securing the Indexer/Forwarder communication channel:

openssl x509 -req -in server.csr -CA /opt/internalCA/rootCA.pem -CAkey /opt/internalCA/rootCA.key -CAcreateserial -out server.crt -days 365 -sha256

Securing the Search Head GUI interaction:

openssl x509 -req -in serverWeb.csr -CA /opt/internalCA/rootCA.pem -CAkey /opt/internalCA/rootCA.key -CAcreateserial -out serverWeb.crt -days 365 -sha256 -extfile csr_details.txt -extensions req_ext

Securing Forwarder to Indexer Communication

Assumes an all in one (AIO) installation. Notes pertain to distributed environments.

  1. Navigate over to /opt/splunk/etc/auth, and make a new set of directories
    cd /opt/splunk/etc/auth && mkdir -p TLS/{idx,for} && cd TLS
  2. Generate the certificate signing request (csr) and keys for the indexer
    openssl req -nodes -newkey rsa:2048 -keyout idx/splunkIDX.key -out idx/splunkIDX.csr
    Fill out the information about the server, but leave the Challenge Password blank (just press enter)
  3. Generate the certificate signing request (csr) and keys for the forwarders
    openssl req -nodes -newkey rsa:2048 -keyout for/splunkFOR.key -out for/splunkFOR.csr
    No need to fill anything out, just hold enter to blank the fields out.
Directory Structure So Far

In a production environment, you’d send the csr files over to the certificate authority, externally or internally hosted, and request a signing. In this case, we’ll use our own certificate authority to sign the csr files.

  1. Run the CA command to sign certificate requests against the splunkFOR and splunkIDX csr files.
## Indexer:
openssl x509 -req -in idx/splunkIDX.csr -CA /opt/internalCA/rootCA.pem -CAkey /opt/internalCA/rootCA.key -CAcreateserial -out idx/splunkIDX.crt -days 365 -sha256## Forwarder: 
openssl x509 -req -in for/splunkFOR.csr -CA /opt/internalCA/rootCA.pem -CAkey /opt/internalCA/rootCA.key -CAcreateserial -out for/splunkFOR.crt -days 365 -sha256

With our certificates now signed, and valid for 1 year (365 days), we can encrypt our keys, and begin to concatenate files into chains.

  1. Create the indexer’s encrypted key. Referenced later as $idxPass$
    openssl rsa -aes256 -in idx/splunkIDX.key -out idx/idxEncrypted.key
  2. Create a new app for your AIO server to utilize
    mkdir -p /opt/splunk/etc/apps/idxTLS/{certs,local}
    **Note: In a clustered environment, you would put these into your distributed apps directory (deployment-apps, master-apps)
  3. Build your certificate file properly in the new directory:
    cat idx/splunkIDX.crt idx/idxEncrypted.key /opt/internalCA/rootCA.pem >> /opt/splunk/etc/apps/idxTLS/certs/idxChain.pem && scp /opt/internalCA/rootCA.pem /opt/splunk/etc/apps/idxTLS/certs/
  4. Fill out a local inputs.conf with the following information
    vi /opt/splunk/etc/apps/idxTLS/local/inputs.conf
# Set the input to listen for Splunk to Splunk on port 9998
disabled = 0# Define the location of the root certificate authority and credentials[SSL]
rootCA = $SPLUNK_HOME/etc/apps/idxTLS/certs/rootCA.pem
serverCert = $SPLUNK_HOME/etc/apps/idxTLS/certs/idxChain.pem
sslVersions = tls,-tls1.0
password = $idxPass$

Restart your Splunk process and confirm the port is open with netstat:

/opt/splunk/bin/splunk restartnetstat -pant | grep 9998

Follow a near identical process for the forwarders

  1. Create the forwarder’s encrypted key. Reference later as $forPass$
    openssl rsa -aes256 -in for/splunkFOR.key -out for/forEncrypted.key
  2. Create a new deployment app for your AIO server to distribute
    mkdir -p /opt/splunk/etc/deployment-apps/forTLS/{certs,local}
  3. Build your certificate file properly in the new directory:
    cat for/splunkFOR.crt for/forEncrypted.key /opt/internalCA/rootCA.pem >> /opt/splunk/etc/deployment-apps/forTLS/certs/forChain.pem && scp /opt/internalCA/rootCA.pem /opt/splunk/etc/deployment-apps/forTLS/certs/
  4. Fill out a deployment outputs and server configuration with the following:

vi /opt/splunk/etc/deployment-apps/forTLS/local/outputs.conf

# Outputs.conf 
defaultGroup = default-autolb-group[tcpout:default-autolb-group]
# server = can be FQDN:port, or IP:Port
server = splunk.home:9998
clientCert = $SPLUNK_HOME/etc/apps/forTLS/certs/forChain.pem
sslPassword = $forPass$

vi /opt/splunk/etc/deployment-apps/forTLS/local/server.conf

# Server.conf
sslRootCAPath = $SPLUNK_HOME/etc/apps/forTLS/certs/rootCA.pem

Push this app to your forwarder, and restart. PS tip: if you need to restart splunkd on a forwarder you don’t have direct access to, you can utilize a blank deployment-app with the restart_splunkd option enabled. Push it out to a target using an existing or new serverclass. Wait a few minutes, and uninstall it. A splunkd restart should kick off.

For a visual/log based representation, search your internal logs for ssl traffic from your hosts:

earliest=@d index=_internal ssl=true 
| stats count by hostname

Securing Search Head Interactions

  1. Navigate over to /opt/splunk/etc/auth, and make a directory to work in
    cd /opt/splunk/etc/auth && mkdir serverWeb && cd serverWeb
  2. Now we’ll generate a file with the certificate request details. This will be referenced later:
    vi csr_details.txt
#This section is stock, do not customize
default_bits = 2048
prompt = no
default_md = sha256
req_extensions = req_ext
distinguished_name = dn#Customization begins here
[ dn ]
C = Country
ST = State
L = Location
O = Organization
OU = Business Unit
emailAddress =
CN = server.domain[ req_ext ]
subjectAltName = @alt_names#This is the important one. If you are running an all in one system, you put the dns and IP down of the system. if you are running multiple search heads in a cluster, put all of their names and IPs down
[ alt_names ]
DNS.1 = server.domain
IP.1 =
  1. Create the signing request utilizing this details file:
    openssl req -new -sha256 -nodes -out server.web.csr -newkey rsa:2048 -keyout server.key -config csr_details.txt
  2. Submit your server.web.csr file to your certificate authority. You should get back a crt, or pem file. If you are running your own certificate authority or lab instance, use the following:
    openssl x509 -req -in server.web.csr -CA /opt/internalCA/rootCA.pem -CAkey /opt/internalCA/rootCA.key -CAcreateserial -out splunkWeb.crt -days 365 -sha256 -extfile csr_details.txt -extentions req_ext
  3. Confirm your returned files show the proper information, including all of your ip/dns entries and alternates:
    openssl x509 -in splunkWeb.crt -text -noout
  4. No we can build the certificate chain
    cat splunkWeb.crt /opt/internalCA/rootCA.pem >> splunkWeb.pem
  5. If you’ve used a password in the creation of the certificate, remove it. Splunk is picky about this.
    openssl rsa -in encrypted.key -out decrypted.key
  6. Create an app to handle all of this (note: in a distributed environment, this would be placed into deployment or shcluster apps, and the paths within the file would reflect the apps/ directory of the target)
    mkdir -p /opt/splunk/etc/apps/webTLS/{certs,local} && scp decrypted.key splunkWeb.pem /opt/splunk/etc/apps/webTLS/certs/ && vi /opt/splunk/etc/apps/webTLS/local/web.conf
enableSplunkWebSSL = true
startwebserver = 1
privKeyPath = $SPLUNK_HOME/etc/apps/webTLS/auth/decrypted.key
caCertPath = $SPLUNK_HOME/etc/apps/webTLS/auth/splunkWeb.pem

Restart Splunk for the certificates to take full effect. Most browsers would be happy without the extra csr details, but Chrome requires it to remove the warnings. With your root certificates already installed on the systems from earlier steps, or inherent in your work, you’ll see something like this:

You may also consider installing the SSL Certificate Checker App from Splunkbase:

It keeps track of your certificates on the system, and can be set up to alert you when you need to renew. Something that isn’t uncommon, even amongst large environments.