Logging IT Data using Splunk – Part 1 – Deploying the Indexer

If you reached this page via search then I guess you know what Splunk is, if not I think I better talk a little about that. Quoting the wiki here:

“Splunk is a search, monitoring and reporting tool for IT system administrators with search capabilities. It crawls logs, metrics, and other data from applications, servers and network devices and indexes it in a searchable repository from which it can generate graphs, SQL reports and alerts. It is intended to assist system administrators in the identification of patterns and the diagnosis of problems. Log files can be correlated across systems and software components which can help administrators uncover the cause analysis of system failures”

The good part about Splunk is that it comes in 2 flavors: enterprise and free. You can get a sense of how the enterprise version works with a 60 days evaluation license. My focus here is on the free edition, probably what most people will work with until they reach the limits of what the free edition can do. The differences between the two versions are presented here on their website.

The latest version is 4.0.x, I have worked with Splunk since version 3.x and I can tell you latest version is a big leap forward. It comes with add-ons that can be piled onto Splunk, called Apps. These apps help you extract information from the data, they make understanding and presenting the data easier. There is also a Windows version of Splunk, that makes forwarding data so much easier.

There are more Splunk deployment models, you can read more about them here. The model I want to talk about is the last one on the list, where we have “Splunk installed on all servers forwarding data“. The below picture from the Splunk wiki is quite self explanatory:

Basically we have 1 installation of Splunk that is the Indexer and we will also install Splunk on each machine we want to index data from,called forwarders. There are 2 types of Forwarders, regular and light. Regular forwarders perform also transformation tasks on the data, sending already tagged information to the Indexer Server, while light forwarders just send the data out to the indexing server with no tagging or transformations applied onto them.

For logging data using Splunk I will show you following:

  • Deploying Splunk as Indexer for Linux
  • Deploying Splunk as Forwarder for Windows.
  • Configuring Forwarders to filter data before forwarding to the Indexer/

Deploying Splunk as Indexer

We are going to install the Indexer on a Linux machine and configure a few forwarders to send data to this machine. There is nothing stopping you from designating a Windows machine for the indexing role though.

For a Debian Linux installation if you copied the .deb file to your linux machine open the console or ssh and run this command:

dpkg -i /tmp/splunk-4.0.8-73243-linux-2.6-intel.deb

Replace "/tmp/splunk-....." with the path to your package. Choose the default settings and when the installation finishes run the command that starts splunk and accepts the license agreement in one step:

/opt/splunk/bin/splunk start --accept-license

Now you can login to the web interface indicated by the installation, by default http://[FQDN]:8000. Default credentials are "admin" with password "changeme". Once there you should see this welcome screen:

Feel free to take a look around, once done find the "Manager" link on the top right-hand side and click it. On the new page find the "Forwarding and Receiving" link. That should point you to a screen looking like this:

Click "Add New" to add a new port on which the Indexing Server will listen for data from forwarders. On the new page insert the port number and save your settings.

There is also the possibility to configure receiving data from the command line.

cd /opt/splunk/bin/
./splunk login

Enter your credentials, then use this command to activate listening on a TCP port:

./splunk enable listen -port <your port number> -auth admin:<password>

You will get this a confirmation message: Listening for Splunk data on TCP port <your port number>.

If you entered the wrong port you can disable listening for that port using:

./splunk disable listen -port <port number> -auth admin:<password>

The confirmation message looks like this: Receiving is disabled on port <your port number>.

At this point you are finished with the basic Indexer configuration. The next post we will cover how to Deploy Splunk Forwarders on Windows machines and get them to send data to the Indexer.