Automate vSphere Certificate Generation

A couple of weeks ago I was working on some audit internally, and I discovered we had some vSphere servers working with self generated certificates. While these servers were un-managed servers (esxi free license servers), they still needed certificates, as it is the case with such servers, they are “critical”, just not critical enough to warrant licenses :).

The “problem’ with vSphere certificates is that they have to be generated using OpenSSL and you cannot generate them using Windows tools like, certreq. With certreq you could potentially have done this process much easier. Also there is an issue with using the request files given out by OpenSSL as it does not have template information written in it, and the Windows CA cannot generate a certificate if it does not know which kind of certificate you want.

I trawled the internet for ways to automate this, and I didn’t find an end to end solution for certificate generation. I only found bits and pieces, and people were writing how to do each certificate one by one. This didn’t sit well with me, and looking at the workflows I discovered there was really no point not having a script that does “it” automatically. I will define what “it’ is, by making a short description of the steps required for generating a vSphere certificate:

  1. Generate CSR file and key file using OpenSSL
  2. Submit CSR file to certification authority
  3. Retrieve response from certification authority
  4. Rename certificate file and key fileΒ  and upload to vSphere host

Some notes regarding the setup in which this would work:

  • I used Powershell to automate this, so this won’t work on other platforms.
  • I used a Windows 2008 R2 PKI CA with a “Web Server” Template.
  • The CA also had automatic approval for this type of certificate (which made automating the response retrieval easier)
  • User running this script needs to have the right to request/issue the given certificate template, also should be local admin on the box you are running the script, otherwise you would have to modify script to run some parts of the commands with “runas”

The script

I used a preexisting script to get started, the one for certificate mass generation from valcolabs.com, found here.

What differs from the way they did it, is that I’ve changed the way variables are passed for building the “config file”,Β  and the fact that each CSR has its own config file, specified on command line. This will help you track your work better for troubleshooting purposes. Something that should be noted is that their script, and also mine, use a special openssl config file, in the sense that the lines to be modified by the script are numbered, not searched in the file, so beware of making changes to the “custom_openssl.cfg” file. It could have probably been more elegant to search for the lines in the file, but I didn’t want to spend time getting it to work.

The download link for the script I built is this one; Generate-vSphere-Cert, below you will find some explanations on how it works.

Learning points

The script takes some parameters as input (get some of them wrong and your script might not work as intended or quit)

a) vSphereHostFile – is a CSV file that must contain the host name and domain name in 2 separate columns.

b) CAMachineName_CAName is the name of your CA in the format (hostname\display name)

c) TemplateName is the name of the certificate template you want to use for certificate generation, as defined on your CA

Lines 32 – 44 you should change the variables there to match your requirements (different paths, different location, country, email, company, etc). There is room for improvement here, you can include this info in the csv file, useful for creating certificates for multiple companies, with different contact information.

Lines 49 – 73 – build out a folder structure, one folder per host where all host files will be stored. Also builds CN, SAN’s (Subject Alternate Names)Β  – you may wish to customize what you add here. I added short name, FQDN, i left out IP address as that can change more easily than the name.

Lines 80-97 – use a temporary file from the original openssl config file containing the parameters we setup until now – this piece of code uses numbered lines, so if you make changes to the original file, change the line numbers here)

Lines 99-104 – build out the file/paths to generate a CSR with openssl. The command i used is slightly different than the ones on the internet, I needed a special length for the RSA, so I used:

"$openssldir\openssl.exe req -newkey rsa:2048 -out $csr -keyout $key -config $config"

Lines 109-114 – build paths for files to send/receive to/from the Windows CA. I also used something “unusual” (as in, not your first page results on google search) which is specifying the CAName and Template name.

The CA name is needed so you do not get a prompt each time certreq is invoked.

The certificate template is specified using the attrib parameter, the missing piece of my “how to automate” CSR submitting, see below:

$ConfigString = """$CAMachineName_CAName"""
$attrib = "CertificateTemplate:$TemplateName"
$issuecerts_cmd = "certreq -submit -attrib $attrib -config $ConfigString $csr $crt $p7b $rsp "

Lines 117-122 Unless you use this script for automating creation of vCenter Certificates, you can comment these lines out. They generate a PFX certificate which is required with vCenter. PFX certificates are not not required for vSphere host certificates.

The next step to automation would be to upload these files to your vSphere host. I used this script here and changed some paths to suit my folder structure. You can also use SCP or other methods to upload the file. After the files are uploaded you need to reboot the host for the certificates to take effect.

As always with these scripts, do your best to try them in a test environment before unleashing them into production. You are dealing with Certification Authorities and your vSphere hosts. Failure to upload a correct certificate to the hosts will result in you not being able to connect with vSphere Client, and having to go to console (NOT SSH) and regenerate self signed certificate.

I hope this was a useful read, comments and critique are open, as always.

Restrict USB Storage Devices on Windows XP

This is one of those topics that are probably on the top 10 to do’s of anyone’s list when it comes to securing their Windows desktops. Whether it is plain dictatorship, security/confidentiality concerns/requirements, unpatched OS’s, weak/no AV solutions, the golden POLP (Principle Of Least Privileges) may force you to come up with a solution to this problem. If you are using anything else (XP, 2000, 2003 Server) except the newer versions of Windows (Vista, 7) which allow you to do this via a GPO setting, you are out of luck, there is no GPO setting or quick-fix that works.

As a short history, I went through CIA documents that were published (can’t find them anymore), Forums, Microsoft KB’s, Whitepapers, and finally came up at the other end of the tunnel with a working process.

The goal is to devise a process of denying access to USB Storage that meets following criteria:

  • Must be implemented at OS level
  • Must be deployed scripted/automatically and/or via GPO
  • Must not cripple other OS functionality (e.g. installing printer/scanner drivers)
  • Must be fully reversible by Administrators only
  • Must be working regardless if USB Storage was used before the process is put in place

The solution – explained

For disabling USB Storage there are 2 situations to cover:

  • No USB storage ever installed, user must not be able to install device
  • USB storage was previously installed by user or admin, user must not be able to use USB Storage again

Both scenarios are covered in these 6 steps:

  1. Copy usbstor.inf, usbstor.pnf, usbstor.sys to their default locations, as if a USB storage device would be installed.
  2. Restrict access to the 3 files mentioned above. We will use an implicit DENY for the local “SYSTEM” Account for these files.
  3. Remove Registry Keys that handle USB Storage device startup: HKLM\SYSTEM\CurrentControlSet\Services\USBSTOR and HKLM\SYSTEM\ControlSet001\Services\USBSTOR and HKLM\SYSTEM\ControlSet002\Services\USBSTOR
  4. Replace USB Storage related registry keys with specially crafted keys that disable startup of the USB Storage driver
  5. Apply an implicit DENY for the local SYSTEM Account on the Registry Keys mentioned above
  6. Insert USB Storage device, wait for it to be detected by OS and marvel at the fact it won’t let you install the device πŸ™ πŸ™‚

For enabling USB Storage these steps must be taken from an Account that is member of the Administrators Group

  1. Remove restrictions placed on the ubstor.* files.
  2. Remove following specially crafted Registry Keys: HKLM\SYSTEM\CurrentControlSet\Services\USBSTOR and HKLM\SYSTEM\ControlSet001\Services\USBSTOR and HKLM\SYSTEM\ControlSet002\Services\USBSTOR
  3. Remove restrictions placed on the registry keys from above
  4. Delete incompletely installed USB storage devices fron Device Manager and Reboot Computer
  5. Insert USB Storage device, wait for it to be detected by OS/go to device manager and refresh device list and marvel at the fact it works πŸ™‚

Implementation – explained

For implementing this in a scripted manner we will use batch scripting, I’m going for a low level approach, assuming you don’t have vbs / powershell on hand, vbs would be rather complicated anyway and Powershell is not installed by default on the OS. You do have some prerequisites:

  • reg.exe (available by default on XP)
  • A network share
  • set-acl (open source utility – get it, copy to a network share of choice and be happy it exists)

Disabling USB

  • The 3 usbstor files mentioned earlier, 2 are available by default (usbstor.inf and usbstor.pnf) under %WINDIR%\inf. The 3rd, usbstor.sys, unless a usb storage device was previously installed is not present. Find it under %WINDIR%\Driver Cache\i386\Sp3.cab or the other cab files there. Extract it from the cab file to the network share.
  • The piece of code that disables USB is written below, but requires that set-acl, the specified .txt, .reg, usbstor.sys files be present in the same directory from which it is executed
::Copy ubstor.sys
xcopy /R /H /Y %CD%\usbstor.sys %windir%\system32\drivers

::Secure USBSTOR.* files with ACE (only Local Administrators Full Control, local "SYSTEM" denied Full Control)
SetACL.exe -on "c:\windows" -ot file -actn restore -bckp "%CD%\usbstor_ACL.txt"

::Delete settings related to USBSTOR Service
REG DELETE HKLM\SYSTEM\CurrentControlSet\Services\USBSTOR /f
REG DELETE HKLM\SYSTEM\ControlSet001\Services\USBSTOR /f
REG DELETE HKLM\SYSTEM\ControlSet002\Services\USBSTOR /f

::Add special crafted registry keys
regedit /s "%CD%\disable_usb.reg"

::Secure keys from above with ACE (only Local Administrators Full Control, local "SYSTEM" denied Full Control)
SetACL.exe -on "hklm\SYSTEM" -ot reg -actn restore -bckp "%CD%\HKLM_ControlSet.txt"
  • Line 5 of the code uses a file that contains a specially formatted ACL applicable to the 3 usbstor files. To generate a different ACL, use the syntax below for each file you are interested in. When you are finished you can merge all text files in a single text file and add it to the script.
SetACL.exe -on "c:\windows\inf\usbstor.inf" -ot file -actn list -lst "f:sddl;w:d,s,o,g;s:b" -bckp "%CD%\usbstor_inf_ACL.txt"
  • REG command is used to delete any data that may exist in the specified registry keys (think previous installed USB Storage)
  • Once the Registry is clean of the keys, we then push a customized reg file (find it at the end of the post), that essentially changes this:

USBSTOR driver points to the file you defined (usbstor.sys, that you just set a restrictive ACL on)

DeviceCount equals zero πŸ™‚

DeviceStartUp Type is set to Disabled (more details here)

Other standard settings for that key

  • Line 16 of code, similar to the ACL for USBSTOR Files, configures the security for the registry keys we added. To customize the ACL, change it to your liking then export the ACL using the command below and update the batch code to include it.
SetACL.exe -on "hklm\SYSTEM\CurrentControlSet\Services\usbstor" -ot reg -actn list -lst "f:sddl;w:d,s,o,g;s:b" -bckp "%CD%\HKLM_CurrentControlSet.txt"

Enabling USB

This is just a question of reversing the changes made by the Disabling process. The following piece of code does just that:

::enable inheritance of permissions
SetACL.exe -on "c:\windows\inf\usbstor.inf" -ot file -actn setprot -op "DACL:np;SACL:np"
SetACL.exe -on "c:\windows\inf\usbstor.pnf" -ot file -actn setprot -op "DACL:np;SACL:np"
SetACL.exe -on "c:\windows\system32\drivers\usbstor.sys" -ot file -actn setprot -op "DACL:np;SACL:np"

::clear any non-inherited ACE
SetACL.exe -on "c:\windows\inf\usbstor.inf" -ot file -actn clear -clr "dacl,sacl"
SetACL.exe -on "c:\windows\inf\usbstor.pnf" -ot file -actn clear -clr "dacl,sacl"
SetACL.exe -on "c:\windows\system32\drivers\usbstor.sys" -ot file -actn clear -clr "dacl,sacl"

::deleting custom Registry Keys
REG DELETE HKLM\SYSTEM\CurrentControlSet\Services\USBSTOR /f
REG DELETE HKLM\SYSTEM\ControlSet001\Services\USBSTOR /f
REG DELETE HKLM\SYSTEM\ControlSet002\Services\USBSTOR /f
  • As you can see we are enabling inheritance of permissions, clearing any ACE defined explicitly on that object (the ones we pushed actually) and removing the Registry keys we also pushed. Make sure the user running this enabling process has rights to change these objects (in our case he is member of the Local Administrators Group)
  • After this is done manually clean it of any hidden installed USB Storage devices and reboot the computer. After the reboot replugging the device should allow you to install and use it again.

Phew, this was also a long post, but believe me, reaching this compressed format was a lot of work :).

Now I’ve attached this zip file containing the contents of what I’ve been talking about, it should be usable out of the box.

There is also there question I guess of securing these files so that they apply to users but users can’t get to them to “help themselves”, but that is another topics for another post perhaps.

As always any feedback is welcomed.