It has been a while since I managed to do some writing on my blog, mostly because I’ve been busy with other Real Life events, and general lack of time. But now I’m here to share something that has been sitting in my drafts folder for a while. This one is about virtualization.
2010 and 2011 were virtualization years for me, I worked on several projects in design, implementation, and I learned so much, that looking back I really get a feeling of accomplishment.
I’ve also been a little “cutting edge”, non conservative with my designs some would say. I guess practice what you preach kind of stuck with me and I made it my mission to build reliable, self contained VMware environments, as much as possible.
As part of the design process, you always have to think about your management software
- Where do you put the pieces of software that help you manage the environment?
- How do you ensure availability and SLA for these components to allow you to recover from failures?
The answer to the first question can be:
Option A: In a management cluster, dedicated to management software for the virtualization stack
- The advantage is you always know where the VMs are, if you have a failure there are 2 servers they can be on.
- The disadvantage is you dedicate two physical boxes for this purpose, which can have a maximum utilization of around 40% for failover reasons.
Option B: Next to production machines
- The advantage is you don’t have to setup a management cluster, and you optimize resource utilization in your datacenter.
- The disadvantage is that you lose “determinism”, the security of “I know on what server vCenter is sitting, so i don’t have to look for it”, if i get a cluster failure or worse.
Well I’ve come up with two “tricks” that tackle the drawbacks of the second option, not knowing where your management servers are, making it a preferred choice if your environment does not warrant a dedicated management cluster just for that.
#1 Track the movement of vCenter and vCenter DB using vCenter Alarms
This one is a really easy way to keep track of your vCenter components. It works best combined with the second trick you will see below, mainly because it does not cover all scenarios but the advantage of this method is that the information is provided in real time.
What I am proposing is that you create an alarm in vCenter, that monitors for events that change the VMhost of your vCenter VM. These events are:
- VM is being migrated (manually)
- VM is being migrated by DRS
- VM is being restarted by HA on another host
The third trigger will be hit and miss, it stands to reason, that if vCenter is not up to send the mail since it being restarted, you may or may not get the email, or you will get it after the fact. nevertheless it is good to have it there.
Below are the screenshots of how the alarm would look like:
On the advanced field put this condition in:
Then add some notification address or whatever you prefer
Save your alarm, and then try to migrate vCenter and see what happens. You should do this to the vCenter DB server aswell, and any components you feel you should know where they are, for troubleshooting purposes (VUM, Nexus 1000v Supervisor Modules, Management Appliances).
#2 Check the vSphere host where vCenter is running using a scheduled script
Another wasy to check where your vCenter components stay is using a scheduled PowerCLI script that runs once a day and sends you an email where vCenter VM and vCenter database are sitting (which vSphere host)
This script assumes following:
- vCenter VM name in inventory = vCenter VM hostname
- vCenter is using separate database, if you don’t care about that, you can remove the references to the DB.
- vCenter Database name in inventory = vCenter Database hostname or at least a cNAME with this name (e.g. RO-vcenter > RO-vCenter-DB name, and alias in DNS)
You can customize this by entering a CSV file of the names of the vcenter instances and their respective databases.
#version 0.1
#initial release
Add-PSSnapin Vmware.VimAutomation.Core -ErrorAction:SilentlyContinue
Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false
#Write-Host -ForegroundColor Yellow "This script Generates a report detailing which host has the vCenter VM and vCenter DB VM`
#If you wish to cancel Press Ctrl+C,otherwise press Enter"
#Read-Host
#using fqdn because certificates are issued using a FQDN
$vCenter = ('vcenter','vcenter2','vcenter3')
If ($global:DefaultVIServers -ne $null) {
DisConnect-VIServer * -Force -Confirm:$false }
$vCenter | % {Connect-VIServer $_ -NotDefault:$false}
$Report = @()
$vCenters = $global:DefaultVIServers | % {
$row = "" | select vCenterInstance,FrontendVMHost,DBVMName,DBVMHost
$row.vCenterInstance = $_.Name
$row.FrontendVMHost = (get-vm -Name $_.Name.Split(".")[0] -server $_.Name).VMHost
#db is hostname + db
$dbvm = "$($_.Name.Split(".")[0])DB"
$DBVMName = ([System.Net.Dns]::GetHostByName($dbvm)).HostName.Split(".")[0]
$row.DBVMName = $DBVMName
$row.DBVMHost = (get-vm -Name $DBVMName* -server $_.Name).VMHost
$Report += $row
}
$FileDate = get-date -Uformat "%Y%m%d-%H%M%S"
$Path = "c:\temp\vsphere\"
$File = "$FileDate-vCenter-InfraLocation.csv"
$Report | Export-Csv -NoTypeInformation -UseCulture "$Path$File"
$encoding = [System.Text.Encoding]::UTF8
#I made the convoluted out-string construct because the object cannot be serialized"
$ReportBody = $null
$ReportBody += $Report | % { "
`n
`n$($_.vCenterInstance)`n$($_.FrontendVMHost)`n$($_.DBVMName)`n$($_.DBVMHost)"}$Body = "</pre>
<div>I'm the PowerCLI Magic Script. This is the list of your vCenter instances and their locations in the Infrastructure.
`
If you ever lose track of them, this email is the reminder. The latest update is from $FileDate
`
Below is the detailed information about each Instance:
`
`n`n`n`n`n`n$ReportBody`n`n
<table border="`"1`"">
<tbody>
<tr>
<td>vCenterInstance</td>
<td>FrontendVMHost</td>
<td>DBVMName</td>
<td>DBVMHost</td>
</tr>
</tbody>
</table>
</div>
<div>"
Send-MailMessage -Smtpserver smtpserver -From 'admin_vmware@foo.com' -To 'vSphereAdministrators@foo.com' -Body $Body -Bodyashtml -Encoding $encoding -Subject "vCenter Instances List" -Attachments $Path$File
Learning points:
Line 11: This is where you define you vCenter server names, if you have more of them. I had 3 for example.
Line 22 & 27: This is where you perform a get-vm to find out the host where this VM is residing on
The rest of the script is just to cycle through all vCenter instances and create an email that it sends to a given email address.
Perhaps to some people this may seem unnecessary, as they may not have faced major outages, perhaps to some it may seem that these monitoring tricks are not enough to cover monitoring of all ‘outages’ situations, but I find it is not worse than having a separate management cluster, with the added benefit of not having to deal with another separate management cluster.
C&C as always is welcome
.
. The next step involved a “google shovel” to “dig up” how to break up those objects into one big list. Then the plan was to have a list that just had the unique email addresses – ideally. So here’s the “magic”: