Logging IT Data using Splunk – Part 1 – Deploying the Indexer

If you reached this page via search then I guess you know what Splunk is, if not I think I better talk a little about that. Quoting the wiki here:

“Splunk is a search, monitoring and reporting tool for IT system administrators with search capabilities. It crawls logs, metrics, and other data from applications, servers and network devices and indexes it in a searchable repository from which it can generate graphs, SQL reports and alerts. It is intended to assist system administrators in the identification of patterns and the diagnosis of problems. Log files can be correlated across systems and software components which can help administrators uncover the cause analysis of system failures”

The good part about Splunk is that it comes in 2 flavors: enterprise and free. You can get a sense of how the enterprise version works with a 60 days evaluation license. My focus here is on the free edition, probably what most people will work with until they reach the limits of what the free edition can do. The differences between the two versions are presented here on their website.

The latest version is 4.0.x, I have worked with Splunk since version 3.x and I can tell you latest version is a big leap forward. It comes with add-ons that can be piled onto Splunk, called Apps. These apps help you extract information from the data, they make understanding and presenting the data easier. There is also a Windows version of Splunk, that makes forwarding data so much easier.

There are more Splunk deployment models, you can read more about them here. The model I want to talk about is the last one on the list, where we have “Splunk installed on all servers forwarding data“. The below picture from the Splunk wiki is quite self explanatory:

Basically we have 1 installation of Splunk that is the Indexer and we will also install Splunk on each machine we want to index data from,called forwarders. There are 2 types of Forwarders, regular and light. Regular forwarders perform also transformation tasks on the data, sending already tagged information to the Indexer Server, while light forwarders just send the data out to the indexing server with no tagging or transformations applied onto them.

For logging data using Splunk I will show you following:

  • Deploying Splunk as Indexer for Linux
  • Deploying Splunk as Forwarder for Windows.
  • Configuring Forwarders to filter data before forwarding to the Indexer/

Deploying Splunk as Indexer

We are going to install the Indexer on a Linux machine and configure a few forwarders to send data to this machine. There is nothing stopping you from designating a Windows machine for the indexing role though.

For a Debian Linux installation if you copied the .deb file to your linux machine open the console or ssh and run this command:

dpkg -i /tmp/splunk-4.0.8-73243-linux-2.6-intel.deb

Replace "/tmp/splunk-....." with the path to your package. Choose the default settings and when the installation finishes run the command that starts splunk and accepts the license agreement in one step:

/opt/splunk/bin/splunk start --accept-license

Now you can login to the web interface indicated by the installation, by default http://[FQDN]:8000. Default credentials are "admin" with password "changeme". Once there you should see this welcome screen:

Feel free to take a look around, once done find the "Manager" link on the top right-hand side and click it. On the new page find the "Forwarding and Receiving" link. That should point you to a screen looking like this:

Click "Add New" to add a new port on which the Indexing Server will listen for data from forwarders. On the new page insert the port number and save your settings.

There is also the possibility to configure receiving data from the command line.

cd /opt/splunk/bin/
./splunk login

Enter your credentials, then use this command to activate listening on a TCP port:

./splunk enable listen -port <your port number> -auth admin:<password>

You will get this a confirmation message: Listening for Splunk data on TCP port <your port number>.

If you entered the wrong port you can disable listening for that port using:

./splunk disable listen -port <port number> -auth admin:<password>

The confirmation message looks like this: Receiving is disabled on port <your port number>.

At this point you are finished with the basic Indexer configuration. The next post we will cover how to Deploy Splunk Forwarders on Windows machines and get them to send data to the Indexer.

Get Forwarding Email Address of Forwarded Mailboxes

This second article, let’s start by talking a little about the “-Filter” parameter, used in most Exch Shell cmd-lets that I’ve found myself abusing of these past days. I have supporting role in an Exchange 2007 Migration Project, and we were tasked with “cleaning up” what is left of the Legacy Exchange Servers in terms of mailboxes. The Domain structure contains multiple forests, one of which has quite a few child domains. In “everything is a possibility” domain structures say you may be required to

Task #1 Get a list of all legacy mailboxes that have forwarding addresses enabled or that DO NOT have a forwarding address at all.

and then

Task #2Based on the list from 1, create a report with forwarded email address and the forwarding address.

Task #1Is done via this one liner

$LegacyFWList = get-mailbox -ResultSize unlimited -RecipientTypeDetails LegacyMailbox -IgnoreDefaultScope -Filter {(ForwardingAddress -like '*')} | select-object ForwardingAddress,WindowsEmailAddress,RealFWAddress

Code Breakdown:

get-mailbox -ResultSize unlimited -RecipientTypeDetails LegacyMailbox -IgnoreDefaultScope -Filter {(ForwardingAddress -like '*')} -ResultSize unlimited
  • By default the cmdlet returns only the first 1000 results. I used unlimited but it also accepts a number as input
  • get-mailbox cmdlet extracts only “LegacyMailbox” objects – this means mailboxes on servers pre Exchange 2007.
  • IgnoreDefaultScope performs a forest-wide search. Without this only legacy MBX’s on exchange servers in the forest root domain would have been returned.
  • Using this does have some limitations to it, check the help for what those are.
  • Biggest limitation is that –Identity must now be the a valid DN of the object.
  • I did not want to extract all DN’s in the forest and pipe them to the cmdlet, I looked for an alternative to this.

I discovered the –Filter parameter. It allows you to apply filters to the search on more Object Properties, much better than –Identity which we cannot use because of the –IgnoreDefaultScope.

A great thing about it is that it can be used without any limitations with IgnoreDefaultScope. Full details about how to use –Filter Parameter and what properties of AD objects are filterable can be found here and here. Back to our task, you can see what we did.

-Filter {(ForwardingAddress -like '*')}

We looked for any objects that contained any value in the “ForwardingAddress” Property of the returned object

    But say you want the opposite of this “get all legacy mailboxes that have no forwarding address”…if would stand to reason that this should work:

    -Filter {(ForwardingAddress -notlike '*')}

    You are out of luck, it returns the entire list of legacy recipients. I went through a few filters until I found this:

      -Filter {(ForwardingAddress -ne $null)}


      -Filter {(ForwardingAddress -ne*')}

      I have to say that it was a surprise when I stumbled upon this, as it doesn’t really make sense. TechNet articles say –like, and I assume also –notlike accept wildcards, while –eq does not, you can also assume -ne does not. Well -ne does take wildcards, which I particularly do not mind in this case. I’m not sure if this is a bug, a feature, or just a flaw in my logic somehow and it’s working because it is supposed to.

      Moving on…

        select-object ForwardingAddress,WindowsEmailAddress,RealFWAddress
        • I can save you running a “| get-member” on what the get-mailbox returned previously and tell you that only “ForwardingAddress,WindowsEmailAddress” are properties returned by get-mailbox.
        • RealFWAddress” is something I made up. Why? Well unless you know already ForwardingAddress is not an address actually, it is a collection of properties for the recipient where emails are forwarded, the list contains DN, ObjectGUID, CN and others, not an actual email address.
        • RealFWAddress is a blank property, attached to the $LegacyFWList object, as a placeholder for when we do script a way to get that email address…which is now! read on

        Task #2 – Create a list of forwarded addresses and forwarding addresses.

        Now we come to the second part of my post, the “get-recipientcmdlet. Since there is no way for us to know to what object are the emails forwarded to [it could be a DL, a contact, a mailbox, another legacy user] we have to make our search as broad as possible. If you try to run get-mailbox on a contact object you will get nothing obviously. We are using foreach-object to go through the entire list. ForwardingAddress is actually the CN of the object. We pass the DN of the Property to a cmdlet that will return a primary email address, if any.

        This following code will accomplish just that:

        $LegacyFWList | foreach-object {
         $ForwardingDN = $_.ForwardingAddress.DistinguishedName
         $Filter = "DistinguishedName -like '$ForwardingDN'"
         $_.RealFWAddress = get-recipient -Filter $Filter -IgnoreDefaultScope | select-object PrimarySmtpAddress}
        $LegacyFWList | export-csv "c:\Change\This\Path.csv"

        Code Breakdown:

        $ForwardingDN = $_.ForwardingAddress.DistinguishedName
        $Filter = "DistinguishedName -like '$ForwardingDN'"
        • I used these 2 variables in the code, $DistinguishedName and $Filter, trying to write a one liner resulted in somehow “deformed” filter command that get-recipient would not understand.
        $_.RealFWAddress = get-recipient -Filter $Filter -IgnoreDefaultScope | select-object PrimarySmtpAddress
        • Lastly, RealFWAddress property I introduced above gets populated with PrimarySmtpAddress value.
        $LegacyFWList | export-csv "c:\Change\This\Path.csv"
        • In the end all that’s left is export the $LegacyFWList variable to CSV and process however you wish, or just store it in xml / pass it on to another part of a script.

        I have to admit the last part of the code is less ideal. Why? Well, the export does not output a clean forwarding email address. I have tried various ways to fix it, but in the end I concluded that it took me 30 minutes of trying to get a nicer export, and 15 seconds to fix it after importing the csv in excel.

        UPDATE 1: 26.01.2010 – added proper way to filter people with no forwarding address
        UPDATE 1: 21.02.2010 – added syntax highlighting and small rewrites

        Exchange 2007 Management Shell – Basics, Tips and Tricks

        This first post i’d like to talk a little about Exchange 2007’s Management Shell. If you are already working with Exchange 2007, you must know by now that there has been a shift in how administration and management tasks are done. Microsoft introduced the Management Shell of Exchange as a Powershell Extension, and a lot of the GUI tasks are now done from the command line.

        Powershell is Microsoft’s new (or maybe not so new, let’s just say latest , it has been with use for some years) extensible command shell interface. It is built on .NET Framework, so it may look more like a shell aimed at the programming savvy, but I think people that have really basic programming skills can handle it after some getting used to.

        How to get it? Well assuming you have an Exchange 2007 server in your organization, you have it by default on the Exchange Servers. You can and should use the shell on the Exchange Servers [running scripts closest to the “managed entities” is always best IMO, you avoid adding additional Points of Failure] but you can also download the Exchange Management Tools for Exchange 2007 and install them on your own workstation, or a workstation you designate for management.

        Ok, so now you have the tools installed. Unless you already found the link to the Exchange Management Shell on your start menu, you can start the tools by typing this in the run box.

        %windir%\system32\windowspowershell\v1.0\powershell.exe -PSConsoleFile "%ProgramFiles%\Microsoft\Exchange Server\bin\exshell.psc1" -noexit -command ". '%ProgramFiles%\Microsoft\Exchange Server\bin\Exchange.ps1'"

        You can probably use this in some scripted NT batch files, if you like. You will find some good information on Google about introductions to powershell, I won’t bore you with them. I’m going to assume you have the shell, have done some really light reading on the topic.

        Here are some quick fixes for Powershell’s quirks though. You must have stumbled upon this at least once.

        Ever ran an extended help command like the one below?

        get-help Get-MailBox -full

        Lots of lines scrolling, you couldn’t read a thing. It’s like one of those days you wish you were either in The Matrix or Startrek, so you can read the damn command-let help slowly or be an Android…or you could have not skipped the RTFM part of powershell, but if you’re like me, you did.

        Worst part about this is that the powershell.exe line buffer is the same as the buffer of cmd.exe, so if the help is big enough, scrolling back up won’t give you a change to read all of the help.

        How to Fix it? – Not to worry old NT batch and Powershell have the answer

        Tip #1 – Using Powershell – For this I should have done the obvious, meaning taking a closer look at the output of:

        get-help get-help (yes you can get-help on get-help)

        There’s a little line that says there’s also the option of running, help [command] instead of get-help. You you try:

        help get-mailbox -full

        You get the equivalent of the /p switch on the NT batch, now that’s help for non Android people 🙂

        Tip #2Oldschool, using NT batch you can redirect the output of help to a text file and read it at your leisure.

        get-help Get-mailbox-full > c:\get-mailbox.help
        notepad C:\get-mailbox.help

        Which one is better?

        Powershell method is quicker because you don’t have to notepad the text file and search for it there. However you cannot use the same method for cmd-let output [or may I have not figured out how, yet], basically reading the output of a cmd-let page by page.

        For that you still have to do use the general method, in NT Batch redirect the output to a file and read that later on, like this for example:

        get-excommand | select-object Type,Name | ft > c:\ex-command.help
        notepad C:\ex-command.help

        This should help anyone struggling with learning the Exchange shell syntax for referencing all of the cmd-let switches available.

        Happy Scripting, hope this first post was not so bad.