The “problem’ with vSphere certificates is that they have to be generated using openSSL and you cannot generate them using Windows tools like, certreq. With certreq you could potentially have done this process much easier. Also there is an issue with using the request files given out by openssl as it does not have template information written in it, and the Windows CA cannot generate a certificate if it does not know which kind of certificate you want.

I trawled the internet for ways to automate this, and I didn’t find an end to end solution for certificate generation. I only found bits and pieces, and people were writing how to do each certificate one by one. This didn’t sit well with me, and looking at the workflows I discovered there was really no point in not having a script that does “it” automatically. I will define what “it’ is, by making a short description of the steps required for generating a vSphere certificate:

1. Generate CSR file and key file using OpenSSL
2. Submit CSR file to certification authority
3. Retrieve response from certification authority
4. Rename certificate file and key file  and upload to vSphere host

Some notes regarding the setup in which this would work:

• I used Powerhsell to automate this, so this won’t work on other platforms.
• I used a Windows 2008 R2 PKI CA with a “Web Server” Template.
• The CA also had automatic approval for this type of certificate (which made automating the response retrieval easier)
• User running this script needs to have the right to request/issue the given certificate template, also should be local admin on the box you are running the script, otherwise you would have to modify script to run some parts of the commands with “runas”

The script

I used a preexisting script to get start, the one for certificate mass generation from valcolabs.com, found here.

What differs from the way they did it, is that I’ve changed the way variables are passed for building the “config file”,  and the fact that each CSR has its own config file, specified on command line. This will help you track your work better for troubleshooting purposes. Something that should be noted is that their script, and also mine, use a special openssl config file, in the sense that the lines to be modified by the script are numbered, not searched in the file, so beware of making changes to the “custom_openssl.cfg” file. It could have probably been more elegant to search for the lines in the file, but I didn’t want to spend time getting it to work.

The download link for the script I built is this one; Generate-vSphere-Cert, below you will find some explanations on how it works.

Learning points

The script takes some parameters as input (get some of them wrong and your script might not work as intended or quit)

a) vSphereHostFile – is a CSV file that must contain the host name and domain name in 2 separate columns.

b) CAMachineName_CAName is the name of your CA in the format (hostname\display name)

c) TemplateName is the name of the certificate template you want to use for certificate generation

Lines 32 – 44 you should change the variables there to match your requirements (different paths, different location, country, email, company, etc). There is room for improvement here, you can include this info in the csv file, useful for creating certificates for multiple companies, with different contact information.

Lines 49 – 73 – build out a folder structure, one folder per host where all host files will be stored. Also builds CN, SAN’s (Subject Alternate Names)  – you may wish to customize what you add here. I added short name, fqdn, i left out IP address as that can change more easily than the name.

Lines 80-97 – use a temporary file from the original openssl config file containing the parameters we setup until now – this piece of code uses numbered lines, so if you make changes to the original file, change the line numbers here)

Lines 99-104 – build out the file/paths to generate a CSR with openssl. The command i used is slightly different than the ones on the internet, I needed a special length for the RSA, so I used:

"$openssldir\openssl.exe req -newkey rsa:2048 -out $csr -keyout $key -config $config"

Lines 109-114 – build paths for files to send/receive to/from the Windows CA. I also used something “unusual” (as in, not your first page results on google search) which is specifying the CAName and Template name.

The CA name is needed so you do not get a prompt each time certreq is invoked.

The certificate template is specified using the attrib parameter, the missing piece of my “how to automate” CSR sumitting, see below:

$ConfigString = """$CAMachineName_CAName"""
$attrib = "CertificateTemplate:$TemplateName"
$issuecerts_cmd = "certreq -submit -attrib $attrib -config $ConfigString $csr $crt $p7b $rsp "

Lines 117-122 - Unless you use this script for automating creation of vCenter Certificates, you can comment these lines out. They generate a pfx certificate which is required with vCenter. PFX certificates are not not required for vSphere host certificates.

The next step to automation would be to upload these files to your vSphere host. I used this script here and changed some paths to suit my folder structure. You can also use SCP or other methods to upload the file. After the files are uploaded you need to reboot the host for the certificates to take effect.

As always with these scripts, do your best to try them in a test environment before unleashing them into production. You are dealing with Certification Authorities and your vSphere hosts. Failure to upload a correct certificate to the hosts will result in you not being able to connect with vSphere Client, and having to go to console (NOT SSH) and regenerate self signed certificate.

I hope this was a useful read, comments and critique are open, as always.

2010 and 2011 were virtualization years for me, I worked on several projects in design, implementation, and I learned so much, that looking back I really get a feeling of accomplishment.
I’ve also been a little “cutting edge”, non conservative with my designs some would say. I guess practice what you preach kind of stuck with me and I made it my mission to build reliable, self contained VMware environments, as much as possible.

As part of the design process, you always have to think about your management software

• Where do you put the pieces of software that help you manage the environment?
• How do you ensure availability and SLA for these components to allow you to recover from failures?

The answer to the first question can be:

Option A: In a management cluster, dedicated to management software for the virtualization stack

• The advantage is you always know where the VMs are, if you have a failure there are 2 servers they can be on.
• The disadvantage is you dedicate two physical boxes for this purpose, which can have a maximum utilization of around 40% for failover reasons.

Option B: Next to production machines

• The advantage is you don’t have to setup a management cluster, and you optimize resource utilization in your datacenter.
• The disadvantage is that you lose “determinism”, the security of “I know on what server vCenter is sitting, so i don’t have to look for it”, if i get a cluster failure or worse.

Well I’ve come up with two “tricks” that tackle the drawbacks of the second option, not knowing where your management servers are, making it a preferred choice if your environment does not warrant a dedicated management cluster just for that.

#1 Track the movement of vCenter and vCenter DB using vCenter Alarms

This one is a really easy way to keep track of your vCenter components. It works best combined with the second trick you will see below, mainly because it does not cover all scenarios but the advantage of this method is that the information is provided in real time.

What I am proposing is that you create an alarm in vCenter, that monitors for events that change the VMhost of your vCenter VM. These events are:

1. VM is being migrated (manually)
2. VM is being migrated by DRS
3. VM is being restarted by HA on another host

The third trigger will be hit and miss, it stands to reason, that if vCenter is not up to send the mail since it being restarted, you may or may not get the email, or you will get it after the fact. nevertheless it is good to have it there.

Below are the screenshots of how the alarm would look like:

On the advanced field put this condition in:

Then add some notification address or whatever you prefer

Save your alarm, and then try to migrate vCenter and see what happens. You should do this to the vCenter DB server aswell, and any components you feel you should know where they are, for troubleshooting purposes (VUM, Nexus 1000v Supervisor Modules, Management Appliances).

#2 Check the vSphere host where vCenter is running using a scheduled script

Another wasy to check where your vCenter components stay is using a scheduled PowerCLI script that runs once a day and sends you an email where vCenter VM and vCenter database are sitting (which vSphere host)

This script assumes following:

1. vCenter VM name in inventory = vCenter VM hostname
2. vCenter is using separate database, if you don’t care about that, you can remove the references to the DB.
3. vCenter Database name in inventory = vCenter Database hostname or at least a cNAME with this name (e.g. RO-vcenter > RO-vCenter-DB name, and alias in DNS)

You can customize this by entering a CSV file of the names of the vcenter instances and their respective databases.

 #version 0.1
#initial release

Add-PSSnapin Vmware.VimAutomation.Core -ErrorAction:SilentlyContinue
Set-PowerCLIConfiguration -DefaultVIServerMode multiple -Confirm:$false
#Write-Host -ForegroundColor Yellow "This script Generates a report detailing which host has the vCenter VM and vCenter DB VM`
#If you wish to cancel Press Ctrl+C,otherwise press Enter"
#Read-Host

#using fqdn because certificates are issued using a FQDN
$vCenter = ('vcenter','vcenter2','vcenter3')

If ($global:DefaultVIServers -ne $null) {
	DisConnect-VIServer * -Force -Confirm:$false }
$vCenter | % {Connect-VIServer $_ -NotDefault:$false}

$Report = @()
$vCenters = $global:DefaultVIServers | % {
	$row = "" | select vCenterInstance,FrontendVMHost,DBVMName,DBVMHost
	$row.vCenterInstance = $_.Name
	$row.FrontendVMHost = (get-vm -Name $_.Name.Split(".")[0] -server $_.Name).VMHost
	#db is hostname + db
	$dbvm = "$($_.Name.Split(".")[0])DB"
	$DBVMName = ([System.Net.Dns]::GetHostByName($dbvm)).HostName.Split(".")[0]
	$row.DBVMName = $DBVMName
	$row.DBVMHost = (get-vm -Name $DBVMName* -server $_.Name).VMHost
	$Report += $row
}

$FileDate = get-date -Uformat "%Y%m%d-%H%M%S"
$Path = "c:\temp\vsphere\"
$File = "$FileDate-vCenter-InfraLocation.csv"
$Report | Export-Csv -NoTypeInformation -UseCulture "$Path$File"

$encoding = [System.Text.Encoding]::UTF8
#I made the convoluted out-string construct because the object cannot be serialized"
$ReportBody = $null
$ReportBody += $Report | % { "
`n
`n$($_.vCenterInstance)`n$($_.FrontendVMHost)`n$($_.DBVMName)`n$($_.DBVMHost)"}$Body = "</pre>
<div>I'm the PowerCLI Magic Script. This is the list of your vCenter instances and their locations in the Infrastructure.
`
If you ever lose track of them, this email is the reminder. The latest update is from $FileDate
`
Below is the detailed information about each Instance:
`
`n`n`n`n`n`n$ReportBody`n`n
<table border="`&quot;1`&quot;">
<tbody>
<tr>
<td>vCenterInstance</td>
<td>FrontendVMHost</td>
<td>DBVMName</td>
<td>DBVMHost</td>
</tr>
</tbody>
</table>
</div>
<div>"

Send-MailMessage -Smtpserver smtpserver -From 'admin_vmware@foo.com' -To 'vSphereAdministrators@foo.com' -Body $Body -Bodyashtml -Encoding $encoding -Subject "vCenter Instances List" -Attachments $Path$File

Learning points:

Line 11: This is where you define you vCenter server names, if you have more of them. I had 3 for example.

Line 22 & 27: This is where you perform a get-vm to find out the host where this VM is residing on

The rest of the script is just to cycle through all vCenter instances and create an email that it sends to a given email address.

Perhaps to some people this may seem unnecessary, as they may not have faced major outages, perhaps to some it may seem that these monitoring tricks are not enough to cover monitoring of all ‘outages’ situations, but I find it is not worse than having a separate management cluster, with the added benefit of not having to deal with another separate management cluster.

C&C as always is welcome

$InputObject = Read-host -Prompt "Insert Computername to get list of installed updates";
$Report = @()
$filename = "$env:Temp\Report_$(get-date -Uformat "%Y%m%d-%H%M%S").csv"
$InputObject | % {
   $objSession = [activator]::CreateInstance([type]::GetTypeFromProgID("Microsoft.Update.Session",$_))
   $objSearcher= $objSession.CreateUpdateSearcher()
   $HistoryCount = objSearcher.GetTotalHistoryCount
   $colSucessHistory = $objSearcher.QueryHistory(0, $HistoryCount)
   Foreach($objEntry in $colSucessHistory | where {$_.ResultCode -eq '2'}) {
       $pso = "" | select Computer,Title,Date
       $pso.Title = $objEntry.Title
       $pso.Date = $objEntry.Date
       $pso.computer = $_
       $Report += $pso
       }
   $objSession = $null
   }
}
$Report | where { $_.Title -notlike 'Definition Update*'} | Export-Csv $filename -NoTypeInformation -UseCulture
ii $filename

Once you run this report and have an csv viewer installed (excel for example( it will open up the file so you can review it. When exporting I did a filter to remove MS Forefront definition updates as it is pretty irrelevant most of the time, you use other tools to manage Forefront definitions.

Learning Points

Line 05 – This line creates and instance of the Windows Update API. What is neat about this function is the fact that can create an instance of the API and connect to a remote computer, notice the “$_” at the end of the line.

Line 09 – In this line after searching the entire history we filter out all but successful updates. Yes it would be nice to do that in the actual search, but I don’t know if it is possible. So I resorted to filtering out only successful result codes.

Below is a table with possible values. This can be useful if you want to generate a report based on the result code

That’s about it with getting the list of installed updates, the bit of code above can be easily integrated to run across a large number of computers. Thanks for reading and feedback.

Now back to the problem at hand. Since the 2 organizations to whom the 2 forests belonged to had pretty loose security requirements, we wanted to save us the hassle of creating and managing a second KMS server, and just using the KMS server we had available. KMS is also not so restrictive when it comes to accepting license validation requests.

KMS clients have 2 ways in which they determine where the KMS service is located (i will use contoso.com as the “main” forest for this example):

• Specify it manually using a built in windows command line script. For example to specify the kms.contoso.com server for a machine just run from administrator command prompt this command:

cscript %windir%\system32\slmgr.vbs /skms kms.contoso.com:1688

• Windows uses DNS to determine the KMS servers (pretty much like Windows does to determine which servers offer AD Authentication). When KMS host is installed in creates an SRV record in the DNS in _.tcp.contoso.com. This record looks like this:

ServiceName: _vlmcs

Port: 1688 (default)

Host offering the service: kms.contoso.com

The Final srv record looks like this: _vlmcs._tcp.contoso.com

As you can see there is not so much rocket science in the way a KMS host is published in DNS. Also there is no requirement that the computer trying to validate a license against KMS be joined to a domain. All the computer needs does is a srv DNS query to determine where the KMS licensing host is. Based on this information it talks to KMS and validates the licenses.

So to make sure computers in forest rivnet.org, for example, can find KMS in DNS do following:

1. Create a new A Record for the IP address of the KMS server kms.contoso.com, in rivnet.org DNS, for example kms.rivnet.org

2. Create a new SRV Type record in _tcp.rivnet.org DNS, with following details

ServiceName: _vlmcs

Port: 1688 (default)

Host offering the service: kms.rivnet.org

The Final srv record looks like this: _vlmcs._tcp.rivnet.org

You can do all this by using dnscmd (available in Windows7/2008) run this command:

dnscmd <DNSServerName> /RecordAdd <ZoneName> _vlmcs._tcp.<ForestName> SRV 0 100 1688

3. Test from a client computer that the SRV record is available in DNS, by running this on a command prompt:

nslookup -type=srv _vlmcs._tcp.rivnet.org

You should get an output that points to the DNS record you created in step 1.

4. Test the client computer can validate his license to the KMS host by running this command from an elevated command prompt:

cscript %windir%\system32\slmgr.vbs /ato

cscript %windir%\system32\slmgr.vbs /dli

There should be a line like this:

KMS machine name from DNS: kms.rivnet.org:1688

And that’s it with using KMS from any other forest in your own AD. In short:

1. Add Host record for KMS host

2. Add SRV record for KMS host

3. Attempt activation, verify activation was done using KMS host.

Fastforward 3 months later and a couple of snapshot VMs and I do a usage report on the datastores to notice something I didn’t expect. I used the VMware vCenter reporting features to get a disk usage (which are pretty sweet by the way). I was amazed the report said zero space used for snapshots (although those VMs had snapshots and VDMKs on the datastores). I cycled through the Datastores and found where the snapshots were stored. They were stored on the Datastore where the OS was found, same where the config file was located, then I looked it up in the documentation and found this:

• The default location for snapshots of Virtual Machines is their Working Directory.

• The default Working Directory is the datastore where the Configuration File (.vmx) of the VM is stored.

Wow, that was unexpected, for me at least since that meant I undersized my OS datastore a little. So this question haunted me, ok, how to change this setting in dire situations, when you want to avoid VMs crashing because your datastore is out of space. I then did more research and discovered this:

• Default Working Directory can be changed if you change the VMX file using by adding/changing this line: workingDir=”path/path/”

• Doing so will ALSO change the location of your .vswp file (the swap file created by vSphere) to the location specified by “WorkingDir”

According to this article you can also specify the location of the swap file within the VMX by adding this line: sched.swap.dir = “/vmfs/volumes/Volume1/VM/”. However this setting or adding the workingdir to the configuration file will take effect over the “Store Virtual Machine Swap file in location specified by the Host” option (on the logic that VM settings take precedence over host settings, unless defaults are used for VM – please correct me if wrong)

The consequence of this is that you no longer define swap file storage at host level (which was pretty easy because you have much fewer hosts) instead you define it at VM level (which you may have in the hundreds). Taking this further you’d probably have to use powershell to set this easily…and have this thorougly documented for each VM.

You can see how from something relatively benign changing defaults for Snapshots turns into quite an administrative burden. Then you have to balance administrative burden vs reisizing datastores.

Datastore sizing – revisited

Now with this information the way datastores are sized get a little more complex. Prior to me knowing about this I read what really smart and knowledgeable people had to say about about datastore sizing and it went a little like this:

(Avg VM * #VMs ) * (100% + (Snapshot Allowance) + 10% Reserve)

Snapshot allowance was 10-20%.

Now that is great for datastores that hold the entire VM inside it, I wanted to separate I/O you have to create multiple datastores and each VM can have more than 1 VMDK the math above applies to a single type of Datastore (e.g. for a Dastore for DB vdmks)

(Avg DB VMDK * #VMDKs ) * (100% + (Snapshot Allowance) + 10%Reserve)

In light of my recent discovery about snapshots, the math changes yet again, the sizing would be:

(AvgVMDK * #VMDKs) * (100% + 10% Reserve)

Now assuming you store the VMDK where you store your OS VDMDK sizing this DataStore changes as follows:

(AvgVDMK * #VMDKs) * (100% + 10%Reserve)+(Other Datastores [db,app,log,swap]) + Snapshot Allowance

Where Snapshot Allowance is now sized different:

Snapshot Allowance = (OS Datastore Size + DB/App/Log/Swap Datastore Size) * (10-20%)

In essence if no VMware snapshot defaults are changed and snapshots will be used (they are found in a lot of processes within VMware – backup solutions, VDI, development, patch management of guests) the space occupied by these snapshots is important and it is also important where snapshots consume this space from. Whatever the design, it must include some form or “snapshots space management” to use some fancy words for it. Any comments or different angles on this are welcome as usual.

• 3 Hosts configured in a cluster. After some weeks it was decided that we had to change the IP’s and vLAN , to make room for some other vLANs that needed room to grow.
• No problem, get the new IP’s, talk to the network guys to trunk the ports on the physical hosts and reconfigure switches to make sure that traffic can talk to our vCenter Server.
• Google for how to change the Service Console IP….5 minutes later Google for how to change also the vLAN ID of the Service Console. So for changing the IP and vLAN these are 2 good places to start.
  • Place host in maintenance mode (while still in cluster – we chose to not remove it or delete the cluster since we had resource pools configured)
  • Make all the change (IP, gateway, hosts file)
  • Test settings (ping, nslookup)
  • Now once all hosts are reconfigured properly we update each host hosts file with the updated IP/hostname entries for the other nodes in a cluster.
• Obviously when I took each host out of maintenance mode our clusters would not work, to be expected.
• Now…let’s reconfigure vSphere Cluster since it was not a proper cluster anymore. Reconfigure cluster finishes “Successfully”(task took longer than we expected it to), everything seems great.

Fast forward a few days later, I do a routine configuration check of the systems and our cluster starts to throw “HA agent misconfigured errors”. I discover although I updated the hosts file on vSphere, the OLD ip addresses were still there. I mean there was a mix of the old settings and new settings. I start asking my colleagues if anyone made any changes, but no one had done anything. After some troubleshooting (which included a file level search for files where that IP may be listed on the vSphere host) I concluded this:

“When you try to reconfigure the IP address of a host that is in a cluster, and then you Reconfigure the cluster for HA, somewhere (maybe vCenter DB) information about the IP’s of the hosts is stored, as they were joined to the Cluster initially! Therefore any cluster reconfiguration of hosts with new IP’s will get a mix of old IP and new IP in the /etc/hosts file and possibly Reconfigure for HA Errors”

To fix this, obviously we disabled HA, disbanded the cluster and recreated it back again.

The right way to change vSphere Service Console IP

In light of these issues these are the steps to properly change the IP address of a host:

1. If host is in a cluster, remove it from the cluster.
2. Put host in maintenance mode.
3. Disconnect from vCenter
4. Login to physical (or remote KVM) console and change IP settings. Change the gateway by editing /etc/sysconfig/network so that the GATEWAY line is pointing to your new gateway. Change the IP using these commands.

esxcfg-vswif -i <new IP > -n <new Mask> vswif0
esxcfg-vswitch vSwitch0 -p <port group Name> -v <VLAN ID>
esxcfg-vswif -s vswif0
esxcfg-vswif -e vswif0

5. Ping your reconfigured host to see all is working properly.

6.Rejoin host to the cluster, reconfigure for HA (let HA reconfigure your hosts file instead of manual changing it). Enjoy not having to worry about cluster issues

A colleague of mine also wrote this “interactive script” that prompts you for required information for changing all these settings, I’m a bit LSI (Linux Shell Impaired).

#!/bin/sh
echo "New IP :"
read new_ip
echo "New Mask:"
read new_mask
echo "New Gw:"
read new_gw
echo "New vlan:"
read new_vlan
sed -i "s/`cat /etc/sysconfig/network |grep GATEWAY=|cut -d = -f 2`/$new_gw/g" /etc/sysconfig/network

esxcfg-vswif -i $new_ip -n $new_mask vswif0
esxcfg-vswitch vSwitch0 -p "Service Console" -v $new_vlan

esxcfg-vswif -s vswif0
esxcfg-vswif -e vswif0

I hope you enjoyed the read, and remember:

If you need to change the IP of a host in a cluster….remove it from the cluster first, saves yourself some time and braincells. Comments and critique are welcome, as usual.

Here is what you will need to accomplish this:

• Exchange Server Administrator Rights where the RSGs are located.
• Exchange Management Shell since this entire procedure is best done using powershell scripting.
• Run the script as Administrator to avoid errors due to enabled UAC

The Script

#Dismount DB, Remove, DB, Remove Storage Group
$DB_in_RSG = Get-MailboxDatabase -Server <ENTERSERVER> | where {$_.Recovery -like '*true*'}
Write-Host -ForegroundColor Green "The Database in the RSG ($($DB_in_RSG.Name)) will be Dismounted. Press Ctrl+C to cancel or Enter to continue"
Read-Host
$DB_in_RSG | Dismount-Database -Verbose -Debug

Write-Host -ForegroundColor Green "The Database in the RSG will be Removed. Press Ctrl+C to cancel or Enter to continue"
Read-Host
$DB_in_RSG | Remove-MailboxDatabase -Verbose -Debug

#Now deleting the files of the database
$SG_Path = (Get-StorageGroup $DB_in_RSG.StorageGroup).SystemFolderPath
Write-Host -ForegroundColor Green "The actual DB files will be removed (stored in $SG_Path). Please check previous steps completed successfully before continuing. Press Ctrl+C to cancel or Enter to continue."
Read-Host
get-item $SG_Path | del -Force -Recurse -Verbose -Debug

#Now removing storage group
Write-Host -ForegroundColor Green "The RSG will be Removed. Press Ctrl+C to cancel or Enter to continue"
Read-Host
Remove-StorageGroup -Identity $DB_in_RSG.StorageGroup -Verbose –Debug

Learning Points

There are a few steps you need to do:

1. Grab the RSG (there is only one per server) – stored in the script in $DB_in_RSG
2. You dismount the database from the server using Dismount-Database cmdlet.
3. After disomunting the database you remove the Database using Remove-MailboxDatabase cmdlet. This will just remove the DB from Exchange, the files will remain on the file system.
4. The actual files on the file system get removed using the path from variable $SG_Path. Use del cmdlet with -force -recurse to bypass any confirmation prompts.
5. The last step is removing the Recovery Storage Group from the Exchange. The commandlet for this is Remove-StorageGroup.

That’s about it, pretty easy actually. Although this script is the last in the series I reccomend you actually run this before anything, before mounting the RSG or at least do a check to see if a RSG is already mounted.

Here is what you will need to accomplish this:

• Exchange Server Administrator Rights where the mailbox(es) are located.
• Full mailbox access for exporting emails out of the temp mailbox I was talking about
• Exchange Management Shell since this entire procedure is best done using powershell scripting.
• Obviously free space both on the Database disk aswell as disk space where the PST file will be saved.

The Script

#Parameters Section
param (
 [parameter(Mandatory = $true)]
 [string]$paramRestoredUser
)

cls
#Restore mailbox section
$Filter = "SamAccountName -like '$paramRestoredUser'"
$SourceAlias = Get-Mailbox -Filter $Filter -IgnoreDefaultScope

If ($SourceAlias -eq $null) {
 Write-Host -foregroundcolor Red "No Mailbox for SamAccountName $paramRestoredUser found.Script will quit"
exit
 }

$RSG_DB = Get-MailboxDatabase -Server $SourceAlias.ServerName | where {$_.Recovery -like '*true*'}
if ($RSG_DB -eq $null) {
 Write-Host -ForegroundColor Red "No RSG was found on $($SourceAlias.ServerName). Script will quit now!"
 exit }

$TargetAlias = Get-Mailbox e2k7_Restore_MBX
Write-Host -ForegroundColor Green "Input the date of the restored mailbox:"
$RestoreDate = Read-Host

$TargetFolder = "$($SourceAlias)_$RestoreDate"
Restore-Mailbox -Identity $TargetAlias.Alias -RSGMailbox $SourceAlias.ExchangeGuid -RSGDatabase $RSG_DB -TargetFolder $TargetFolder -BadItemLimit 1000 -Verbose -Debug -Confirm -ValidateOnly

echo "-ValidateOnly switch from the command above is removed. Please check no errors occured above and everything is configured properly. Press Enter to continue CTRL+C to cancel the script!"
Read-Host
Restore-Mailbox -Identity $TargetAlias.Alias -RSGMailbox $SourceAlias.ExchangeGuid -RSGDatabase $RSG_DB -TargetFolder $TargetFolder -BadItemLimit 1000 -Verbose -Debug -Confirm

#building an excluded folders list, just in case we did another restore and the previous content was not deleted
$IncludedFolders = $TargetFolder
$ExclFoldersList = @()
$ExcludedFolders = Get-MailboxFolderStatistics $TargetAlias | where-object { $_.FolderPath -notlike "*$($IncludedFolders)*"} | Select-Object FolderPath | foreach-object {
 $ExclFoldersList += $_.FolderPath }
$ExclFoldersList = ([string]::join(",",$ExclFoldersList)).Replace("/","\")

$UserName = $TargetAlias
$PSTPath = "Q:\UserPersonalFolders"
"Excluded folders list: $ExclFoldersList"
Read-Host
export-Mailbox -Identity $UserName.SamAccountName -BadItemLimit 1000 -DeleteContent $True -PSTFolderPath $PSTPath -ExcludeFolders $ExclFoldersList

The details for the Script

First the script gathers data about the restored mailbox, and from that info the Recovery Storage Group ($SourceAlias). It also needs a target mailbox, where the restored data will be stored ($TargetAlias) (needless to say I did in a “forest friendly way, using -Filter cmdlet looking up the given SamAccountName). The data will be moved from the mailbox in the RSG into the $TargetAlias mailbox. It will be placed in a folder, named with the date of the restored data ($RestoreDate). Restoring the data is done with this command:

Restore-Mailbox -Identity $TargetAlias.Alias -RSGMailbox $SourceAlias.ExchangeGuid -RSGDatabase $RSG_DB -TargetFolder $TargetFolder -BadItemLimit 1000 -Verbose -Debug -Confirm -ValidateOnly

Now we want to export the data from the $TargetAlias into a PST file. This is easy to do via the Export-Mailbox cmdlet. The problem appears when you do for instance multiple exports in the same target mailbox and then a mass export, or just have left old data in the target mailbox. You do get data separated into folders, if your $TargetFolder name is unique, but the Export-Mailbox cmdlet cannot export data based on this information (it can only do some filtering (date,content,subject) and I believe it is resource intensive). What Export-Mailbox does have is the “-Excludefolders” parameter which lets you not export certain data.

The trick I came up with was to scan the $TargetAlias mailbox for foldernames (I used Get-MailboxFolderStatistics for that) and build a list of the folders that did not contain the value I entered in the $TargetFolder variable upon restore. Besides that you also have to do some parsing of the output from get mailboxfolderstatistics into something Export-Mailbox understands. This is what I did:

$IncludedFolders = $TargetFolder
$ExclFoldersList = @()
$ExcludedFolders = Get-MailboxFolderStatistics $TargetAlias | where-object { $_.FolderPath -notlike "*$($IncludedFolders)*"} | Select-Object FolderPath | foreach-object {
 $ExclFoldersList += $_.FolderPath }
$ExclFoldersList = ([string]::join(",",$ExclFoldersList)).Replace("/","\")

In the end I just ran the Export-Mailbox on the target mailbox, specifying the excluded files and that was that.

The whole process is pretty easy, the actual restore is done in a one liner (Restore-Mailbox), but as you can see error checking and failproofing the script make up the rest of the work. Hope you enjoy this, next up is how to dismount a RSG, the scripted “clean-way”.

Q: But why don’t we use the nice GUI Tool from Exchange Management Console (Extra.exe) and do it from there, “we don’t need no scripting”?

A: My experience tells me the scripted method is safer and works “as expected” unlike the GUI, which says it did something, when it didn’t (I’ve spent days trying to figure out why a RSG Database won’t actually dismount when the GUI said: “Completed Successfully”.

OK, let’s get on with it. All that I am about to explain requires Exchange Administrator privileges on the Exchange servers.

We will be creating a Recovery Storage Group, this is the first step in the restore process. To create the RSG you need following:

• Adequate disk-space to restore the mailbox database, locally on the Exchange Server where the DB was residing
• Exchange Management Shell running as Administrator (especially on CCR clusters)
• No other Recovery Storage Group already created on that server with an existing RSG database (you can only have 1 RSG with 1 DB in the RSG). It is best to remove any previous RSG completely then recreate it for your needs.
• Specific information like which DB to link to the RSG and the list of mailboxes to restore.

Creating a Recovery Storage Group can be as easy as this:

New-StorageGroup -Server <MBX Role Server Name> -Name <StorageGroup Name> -LogFolderPath <Logs Folder> -SystemFolderPath <SystemFiles Path> -Recovery -Verbose

The command is very similar to creating a new SG, except for the -Recovery switch, designating it as a Recovery Storage Group. I added the -Verbose switch so you can see what is going on behind the scenes.

New-MailboxDatabase -MailboxDatabaseToRecover <Mailbox Name> -StorageGroup <Recovery Storage Group Name> -EdbFilePath <path to store edb file> -Verbose

Here it is just as easy as creating a new mailbox database, only you are creating it in the recovery storage group you created with the previous command. The key thing to remember here is that the value of the “MailboxDatabaseToRecoverParameter” must be the exact same name of the mailboxDB of which you want to recover from. If the name is different you will not be able to run any restore commands, because it will not be able to find any mailboxes when it searches the recovered database.

A working script for creating the RSG

Below I’m sharing with you a working snippet that should help in creating a recovery storage group and DB. In short here is what the code does:

Using a given UserPrincipalName…

• Attempts to retrieve the mailbox for the UPN (it is a “forest friendly” coding for retrieving the mailbox). If it fails it quits
• Checks if a folder structure for placing, logs, system files and the edb file exists (I used a location called d:, use a variable if you like).
• If folders already exist, it will quit, otherwise it will create a folder with the MDB name, and logs and edb subfolders,
• Next it checks if a Recovery Storage Group already exists, unless you cancel the script it will continue to use this RSG, with the given details. Otherwise it will create a RSG on its own.
• It will then create a mailbox database where you / your backup admin will restore your exchange backup.

$MBX_UPN = Read-Host
$Filter = "UserPrincipalName -like '$MBX_UPN'"
$SourceMBX =  get-mailbox -IgnoreDefaultScope -Filter $Filter
If ($SourceMBX -eq $null) {
	Write-Host -foregroundcolor Red "No Mailbox for $MBX_ID found`nScript will Quit"
	exit }
Write-Host -ForegroundColor Green "Source Mailbox is`n $SourceMBX"

$LinkedMDB = Get-MailboxDatabase -Identity $SourceMBX.Database
Write-Host -ForegroundColor Green "Ok, Database ($($LinkedMDB.StorageGroup.Name)) is grabbed, now creating RSG Folders and RSG`nPress Enter to continue or Ctrl+C to Cancel"
Read-Host

#Checking if the RSG folders already exist, if not attempt to create them
If ((Test-Path "d:\$($LinkedMDB.StorageGroup.Name)")) {
 Write-Host -ForegroundColor Red "Folder already exists. Please remove d:\$($LinkedMDB.StorageGroup.Name) before running this script again.`nScript will quit"
 exit
 }
$SysPath = New-Item -Type Directory -Path d: -Name $LinkedMDB.StorageGroup.Name | Get-Item
If ((Test-path $SysPath)) {
 $DBPath = New-Item -Type Directory -Path $SysPath -Name DB | Get-Item
 $LogsPath = New-Item -Type Directory -Path $SysPath -Name Logs | Get-Item
 }
#If folders were created successfully we can continue
If ((Test-path $SysPath) -and (Test-path $DBPath) -and (Test-path $LogsPath)) {
 #Checking if RSG already exists
$RSG_check = Get-StorageGroup -Server $LinkedMDB.ServerName | where {$_.Recovery -like "True"}
 If ($RSG_check -ne $null) {
 Write-Host -ForegroundColor Magenta "A RSG was found on $($RSG_check.ServerName). Here are RSG Details:"
 $RSG_Check | select-object Name,Identity,Recovery,LogFolderPath,SystemFolderPath | fl
 Write-Host -ForegroundColor Magenta "To use this RSG Press Enter, to cancel Press Ctrl+C"
 Read-Host
 }
 Else {
 Write-Host -ForegroundColor Green "Now creating Recovery Storage Group..."
 New-StorageGroup -Server $LinkedMDB.Server -Name "Recovery Storage Group" -LogFolderPath $LogsPath.FullName -SystemFolderPath $SysPath.FullName -Recovery -Verbose
 }
 Write-Host -ForegroundColor Green "OK! No RSG found. Now creating RSG Database..."
 New-MailboxDatabase -MailboxDatabaseToRecover $LinkedMDB.AdminDisplayName -StorageGroup "$($LinkedMDB.ServerName)\Recovery Storage Group" -EdbFilePath "$($DBPath.FullName)\$($LinkedMDB.Name).edb" -Verbose
 }
Else {
 Write-Host -ForegroundColor Red "Could not Create folder or folder structure in d:\$($LinkedMDB.StorageGroup.Name). Check messages above for errors! Script will quit."
 exit
 }

This is about it with creating a Recovery Storage Group, it is actually not difficult, just remember to name the MDB inside the RSG with the same name as the source MDB (this was also required on Exchange 2003, as far as I know). Also you cannot have more than one RSG per Maibox Server, it is best to remove any RSG you have after you are finished recovering data. Next post we will discuss how to restore data from a MDB and how to remove the RSG.

As always I value your feedback and hope you found this post useful.

The steps below should be attempted after you have backed up your Windows installation and/or relevant documents, please do not attempt this procedure before doing a backup of your system. Double, triple, quadruple check that the problems I am describing here exactly match your hardware, software and symptoms observed. This procedure can damage your operating system, possibly even the hardware, this post comes with no warranties, it is not supported by HP, Intel or any other vendor as far as I know. Also this post is valid at the time of writing, new fixed drivers may appears by the time you are reading this, making it obsolete.

Symptoms and Conditions

• Sluggish Disk performance
• Sluggish computer performance when doing disk based operations
• BIOS is configured to use IDE mode not AHCI mode for SATA disks (you configured for IDE because AHCI was not working)
• Device Manager is showing a primary IDE channel device configured for PIO mode only – you cannot select UDMA mode
• Using SysInternals Process Explorer reveals 25% CPU is Hardware Interrupts when accessing disk continuously (on an i5 cpu that is 1 core…spread over all 4 cores). Interrupts usage goes down when disk is idle.
• Your storage controllers are detected as:
  • “Intel(R) 5 Series/3400 Series Chipset Family 2 port Serial ATA Storage Controller”
  • “Intel(R) 5 Series/3400 Series Chipset Family 4 port Serial ATA Storage Controller”
• The Storage Controllers detected above have Hardware ID’s:
  • PCI\VEN_8086&DEV_3B2D
  • PCI\VEN_8086&DEV_3B2E

Trials and Errors

As you can see starting from the High Hardware interrupts up to device being put into PIO mode, it is clear that there is some driver issue somewhere. What you can try and watch it fail:

• Delete disk controller devices then use scan for hardware changes so windows will reinstall drivers. After reboot you will see the problem still exists.
• Delete disk controller devices then download latest Intel drivers, automatically choose which driver to install. After reboot you will see no changes, same device will be detected, same drivers installed.
• Reinstall OS re-add drivers one by one, you will have the same problem.

The solution

You need to manually select a device driver to install from the driver package for Intel’s Rapid Storage Technology (I think the former name for these drivers, in general, was Matrix Storage Manager). The problem is that the drivers that come with Windows can only use PIO Mode, and the driver package from Intel does not contain the Hardware ID’s you found above. The drivers do work, to get them to work you have to either:

1. Manually install drivers selecting a specific device driver to install
2. Hack the driver files so they include your device ID’s (“hardcore” option, try #1 before you go there )

Option 1 – Do a manual driver install

This works mostly after you installed the operating system. here’s what you need to do exactly:

• Download the driver package
• Using Device Manager delete storage controllers from the laptop
• In Device Manager click “Scan for hardware changes“, the New Hardware Wizard appears
• Choose don’t search Windows Update
• Next choose “Install from specific location“
• Next choose “Don’t search I will choose driver to install“
• On the next screen click on “have disk” and point it to the location of the extracted driver files. Click OK to close driver selection. List will be populated with a bunch of devices
• From the devices list select “Intel(R) 5 Series 6 Port SATA AHCI Controller“
• Click OK and correct drivers should install now. If you are asked to reboot, choose OK
• After the reboot go into the BIOS, change SATA mode to AHCI. If you keep SATA mode to IDE your XP install will BSOD (the reason is you added SATA drivers to XP, and the controller talks IDE, if left unconfigured)
• Now you should see that your devices are installed correctly and you have no more hardware interrupts. Also the Disk Controllers section in Device Manager looks different, fewer devices left there.

Option 2 – Hack the Driver files

This option is useful if you want to make a driver package for an unattended installation or just want to have a set of drivers that will work “out of the box”. What we will do in short is add a few lines of code to the files in the driver package, pointing the Hardware ID’s to the “Intel(R) 5 Series 6 Port SATA AHCI Controller” we manually installed with Option 1. Here’s how to do it:

• Extract drivers to a folder, you should have these files inside among some other txt’s:
  • iaAHCI.cat,iaAHCI.inf,iaStor.cat,iaStor.inf,IaStor.sys,TXTSETUP.OEM
• Open iaAHCI.inf file for editing and search at the end of the file for the “strings” section. Look for the string “PCI\VEN_8086&DEV_3B2F&CC_0106.DeviceDesc” which matches to the Intel 5 series 6 port controller . As you can see after the DEV_ follows “3B2F”, pretty similar to our Hardware ID’s:
  • PCI\VEN_8086&DEV_3B2D
  • PCI\VEN_8086&DEV_3B2E
• Before the DEV_3B2F line create 2 new lines where you duplicate the DEV_3B2F line, BUT you replace 3B2F with the last 4 characters from the other device ID’s (one line will have 3B2D the other 3B2E). The point is to have the Hardware ID’s of your controller point to the correct driver name.
• Now we have to track any place in the document where “3B2F” appears and add the same text for Hardware ID’s 3B2F and 3B2E. The section you are looking for to add lines are in “[INTEL_HDC.ntx86]“, there is a line containing 3b2f, add 1 line for each Hardware ID.
• Save iaAHCI.inf and close it
• Update Disk Controller drivers by pointing Windows hardware wizard to your modifed .inf file
  

With this inf file Windows should be able to install the driver it needs without you having to select which driver to install from the list. The logic is that now Windows knows where to find the correct drivers, because the Hacked Intel Driver contains the device ID’s Windows is looking for.

Option 2+, unattended installs

This next section is about changing the TXTSETUP.OEM so you can do unattended installations using this hacked INF file. You can follow the Intel guide to injecting drivers for “F6 Install”, but you need to change the TXTSETUP.OEM file that comes with this package. Do following:

• From the driver package Open TXTSETUP.OEM for editing.
• In the iaAHCI.inf section look for the “Intel(R) 5 Series 6 Port SATA AHCI Controller”. To the left of that string is the text “iaAHCI_5_1″.
• Do a search for the string “iaAHCI_5_1″ in the document, you should find a section called “[HardwareIds.scsi.iaAHCI_5_1]“.
• When found copy it and the line after it ( looks like “id = “PCI\VEN_8086&DEV_3B2F&CC_0106″,”iaStor”") 2 times. The 2 copies you can change instead of being 3B2F to 3B2D and 3B2E respectively.
• In the end you should have 3 “hardwareIDs” sections, 1 with 3B2F, the original and the other 2 Hardware ID’s you need.
• Save and close TXTSETUP.OEM.
• Follow Intel’s “F6 install” procedure to deploy Windows XP using these modified files (all the files in the package + modded iaAHCI.inf and TXTSETUP.OEM)
• You must configure BIOS to use AHCI mode, drives will not work with IDE mode (didn’t for me)

Phew, this was a long and “hard” post. I hope the general idea is clear:

For installing from windows just make sure to select the controller I mentioned (the 6 port device) when doing the complete manual install.

For the hacked inf and OEM files double check and triple check the changes you are making. the point is to add the HW ID’s to the INF file, so it will install the drivers the same way as for the Hardware ID ending with 3B2F.

My best guess is that this mess-up is due to some slightly different versions or ID’s being stamped erroneous onto the controllers when they were shipped. I hope this was helpful, please report back any mistakes you notice.