Logging Data Using Splunk – Part 2 – Deploying the Forwarder on Windows (continued)

I promised the earlier posts We will talk a bit about how to actually configure a Splunk forwarder once you set it up on Windows. I will try to cover very briefly following:

  1. Configure Data Inputs
  2. Configure Data Output

A few “ground rules”:

  • Everything I will be talking about in this post refers to configuration files found in “<Splunk Install Path>\” folder, referred to as SPLUNK_HOME variable.
  • Configurations are read from SPLUNK_HOME\etc\system\local\ but if these files do not exist/define a specific setting, the corresponding defaults file is loaded, stored in SPLUNK_HOME\etc\system\default\. By default files in the local folder contain minimal data [e.g. host name].
  • Config file formatting : file extension is .conf, comments are prefixed with “#”, a “configuration section” has the title marked between “‘[ ]” and contains data until the next configuration section, or the end of file.

Splunk Data Inputs

Splunk explains the use of data inputs for Windows here. Data inputs are basically sources where Splunk grabs data and forwards/indexes it.

While Splunk can do so many things like WMI queries, AD monitoring, file monitoring and use python searches to index whatever you want it to, I will just show you how to make it listen to Event Logs only, personally I think the other things are stretching it a bit

If Splunk is just a side application to your organization, e.g. you use the free license, you may think his strength is in forensic analysis rather than all sorts of monitoring, which while it is nice, require more knowledge than just these basic posts of mine ;).

If you remember I told you in the last post, that while Splunk does have installation parameters, I prefer pushing the configuration after it was installed, before the first run. The reason is that I noticed the installer doesn’t really do all that it advertises, and since you, like me may want to do customizations to the config files, why not make a baseline/template “config files pack” that you push afterwards to the server.

By default a Windows Splunk install indexes some Event Logs and also uses some WMI queries, so first order of business is to make it “unindex” anything WMI/Script related. Add this to your inputs.conf file.

#Disable registry monitoring and WMI scripts
[script://$SPLUNK_HOME\bin\scripts\splunk-regmon.py]
disabled = 1

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.py]
disabled = 1

# Pull event logs from the local system
# Usually disabled in favor of using WinEventLog inputs
[WMI:LocalApplication]
disabled = 1

[WMI:LocalSystem]
disabled = 1

[WMI:LocalSecurity]
disabled = 1

# Gather performance data from the local system
[WMI:LocalPhysicalDisk]
disabled = 1

[WMI:LocalProcesses]
disabled = 1

[WMI:Memory]
disabled = 1

[WMI:LocalNetwork]
disabled = 1

[WMI:CPUTime]
disabled = 1

[WMI:FreeDiskSpace]
disabled = 1

As you can see it is just a matter of setting a disabled flag. You can get a list of sources to to disable, if you go into the GUI of splunk and enable all monitoring sources then start disabling what you wish.

“Un-indexing” is finished, now we start adding what event-logs we want indexed and also set some parameters for each Event-Log.

[default]
evt_dc_name =<Add FQDN of DC here >
evt_dns_name = <Add FWDN of DC here >

The above section defines a domain controller and a DNS server for resolution of GUID/SID values in the events indexed. It is a global value, applies to all event-log sources.

[WinEventLog:Application]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5

Pretty self explanatory in a way – [WinEventLog:Application] defines which application Log we are looking at. You can change the value that appears after “:” to whatever you want for example: “Directory Service”, “Powershell”,”Microsoft Office”,”Security”,”System”, any event-log name.
We set it to be enabled and to start indexing from the first event in the event log, and index all events, not just the current events (unlike a linux tail command). Evt_resolve_ad_obj enables or disables resolving of GUID/SID. CheckpointInterval is how often it polls for new data coming from the event logs.
Now you can multiply this configuration segment for as many event-logs you have defined on your system. Also if you define an event-log name that does not exist, not a problem, splunk won’t index it, it will just log an error in the event log, personally I can live with that.
Also Splunk can monitor flat files and one flat file you may be interested as a sysadmin is the windowsupdate.log file. Splunk Indexer can have a smart little application called “Windows” that can actually makes sense of those logs for you, to some extent. Here is what you should add to you inputs.conf to get that file to be monitored.

[monitor://$WINDIR\WindowsUpdate.log]
sourcetype = WindowsUpdateLog
disabled = 0

Now save this file in SPLUNK_HOME\etc\system\local\ and start/restart Splunk to see some results, or just hold that thought until the end of the post. If you followed my guide sofar, this should give you actually a Splunk Indexing Server, because all that we did was install it and configure what it indexes. Next step we make it send data out somewhere, which turns it into a forwarder and disables local indexing.

Configure Data Output

For more official Splunk info go here. For a forwarding crash course read on. We will configure Splunk to send data out to “groups”. These groups can be actually a single host, or a group of hosts (think indexing load-balancer configuration). The following will configure forwarding of all events to a host group made out of a single host. The configuration file should be outputs.conf stored in the same location as inputs.conf from before. Add this to the file:

[tcpout]
defaultGroup = &lt;group_name_ID&gt;
disabled = false

[indexAndForward]
index = true/false

[tcpout:&lt;group_name_ID&gt;]
server=&lt;IP&gt;:&lt;port&gt;
heartbeatFrequency=45
maxQueueSize=10000

First section describes the group configuration, that you detailed below and enables it. Next you can forward data and also index it locally. A forwarder does not need to keep data locally, so you set this to False.

The “tcpout: < group_name_ID>” defines a group of settings pointing to a listening server.Group_name_ID from here must match with the value you entered the first time, when you mentioned the Group_name_ID.

Server lets you define the server and listening port. The IP and port must match the IP and listening port of the Indexer.

The heartbeat frequency is basically how often the Indexer is being polled if he is alive.

maxQueueSize is by default 1000, for busy servers you may want to increase it to something more, like I did.

Now you should be done. You can start/restart splunk by starting the splunkd service from the services snap-in or run “net start splunkd” from the command line prompt on the server. If you did everything right, Splunk should be acting as a forwarder now and send data to the indexing server. If you do not see anything on the Indexer, start troubleshooting both the indexer and the forwarder. The logs from “SPLUNK_HOME\var\log\splunk\” may prove useful.

I hope this introduction was helpful to anyone trying to get a basic grasp of a Splunk setup for Windows, I wish you Happy Splunking! 🙂


Print pagePDF pageEmail page
Tagged , , , , . Bookmark the permalink.

Comments are closed.