Automate Replacing of Certificates in vCenter 5.1

A few days ago, VMware launched a much awaited tool, called SSL Certificate Automation Tool. This tool enables VMware administrators to automate the process by which they replace expired/self-signed certificates on all components of the VMware vCenter management suite. As many of you know this process, especially in the new v5.1 version is a complete pain to implement, error prone, and so many steps to follow that you are bound to make a mistake. Compared to say VMware vCenter 4.x, version 5.1 has more “standalone” components that need to interact with users or interact with each other to provide users with data/visualizations and must do that over secure connections.

The components I’m talking about are (the ones highlighted in orange are new to version 5.x vs 4.x)

  • InventoryService
  • SSO
  • vCenter
  • WebClient
  • LogBrowser
  • UpdateManager
  • Orchestrator

It might not look like much but it’s almost double the number of components, double the number of certificates and close to double the number of interactions between the components themselves. To make is “worse”, the workflow for replacing v4.x certificates  (I have this post where I automated the whole process for ESXi 4.x, extendable for vCenter 4.x) is different than the workflow for v5.x ones, much more complicated. Here‘s a document how to manually do it. But if you value your time read on, there’s  an easier way.

The relevant documentation for this automation process is available here from VMware. Essentially it a 2 step process:

Step1: Generate your certificates using OpenSSL and your Internal Windows Enterprise root CA, as per this document(Generating certificates for use with the VMware SSL Certificate Automation Tool (2044696)

Step2: Use the SSL Certificate Automation tool to deploy the certificates, as described here.

Automate Certificate Generation for use with the VMware SSL Certificate Automation Tool

I should say this again, your main source of information should be VMware’s KB article. The information here is either echoing that, or supplementing it where needed. Also things are presented in a different order in my post, so the whole process can be automated, unlike the KB which assumes manual work.

Prerequisites

1. The account you will use in this step must:

  • Be a Local administrator on the computer where the script presented will run
  • Be able to enroll certificates for the Certificate Template that will be used from your PKI infrastructure.
  • Be a Local Administrator on the servers where the vCenter components are installed.

2. Any commands/scripts presented here should be run from an elevated prompt.

3. Name resolution must work correctly on the client where this script will run (all vCenter components must be resolvable via DNS).

4. You must use the OpenSSLversion that VMware specifies, not newer, not older, that is openSSL 0.98 at the time of writing this post.

5. You need to have a certificate template configured according to VMware’s specifications, that means basically duplicating the default web server certificate of the Windows CA, with a few changes:

  • Go to Certificate Manager > Extensions > Key Usage > Allow encryption of user data
  • Also uncheck “allow private key to be exported” as the script will fail if the template allows this.

6. Your CA must have automatic approval activated so you can obtain the certificate using command line.

7. Obtain and save the certificates of your root CA and any intermediate CAs, as described in the KB. From the KB, as I understand it, you should name your root CA certificate root64.cer (x.509 format, base64 encoded). Any other intermediate or issuing CA certificates should be named root64-#.cer (x.509 format, base64 encoded), where # is a number starting from 1 to as many intermediates you have (where root64-1 is the intermediate closest to the issued certificate, and root64-N is the certificate closest to the root CA). Place all the certificates from the chain in a folder called “RootCA_chain“.

Note: Naming the files in a certain way is important. The KB does not do a good enough job of explaining how to build the .PEM file (either that, or I’m not doing a good enough job of understanding it), so make sure your CA chain certificates are numbered like this. if you don’t do this way you will have to rework the piece of code that sorts the certificate files for building the .PEM file.

8. Make yourself a custom openSSL configuration file. The configuration file must include only these lines, not more, not less (it’s a copy paste from VMware’s KB). Save this file as custom_openssl.cfg in the \bin directory where you installed OpenSSL.

[ req ]
default_bits = 2048
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:ServerShortName, IP:ServerIPAddress, DNS:server.contoso.com #examples only

[ req_distinguished_name ] # change these settings for your environment
countryName = US
stateOrProvinceName = Change State
localityName = Change City
0.organizationName = Change Company Name
organizationalUnitName = ChangeMe
commonName = Changeserver.contoso.com

9. Build a .csv file like this example below:

Name,DomainName,OUName
CNTvcS,contoso.com,vCenterServer
CNTvcO,contoso.com,VMwareOrchestrator
CNTvcIS,contoso.com,vCenterInventoryService
CNTVUM,contoso.com,VMwareUpdateManager
CNTwc,contoso.com,vCenterWebClient
CNTwc,contoso.com,,vCenterLogBrowser
CNTsso,contoso.com,vCenterSSO

As you can see, you need to specify a server name, domain name, and OU Name. The OU name is the name of the vCenter component, the value will be written in the “organizationalUnitName” part of the OpenSSl configuration file.

10. Create a folder (the script uses a static c:\temp\certs entry) where you will store all your certificates as they are generated. To this folder copy the RootCA_chain folder created above.

Just like with cooking you now have all the “ingredients” to generate your vCenter certificates.

The Script

You can download the script (Generate_vCenter_Certs_v5.1) from on my blog. My previous post on VMware certificates has a little more background information, that I won’t repeat here.

Learning Points

Lines 16-50: The script takes a few parameters, and also does some error checking on them.

    • The location of the openSSL directory ($OpenSSLPath)
    • The CSV from step 9, with all the servers and components of vcenter ($vCenterHostsFile)
    • The name\display name combination for your issuing/root Certification authority ($CAMachineName_CAName).
    • The name of the template to be used when issuing the certificates ($TemplateName).

Lines 52-71: More variables are set, and the script checks whether you have the root CA files in the specified location (default: c:\temp\certs\RootCA_chain).

Also this is where you should go in the script and change the Country, Company, State, and City Name. These will replace the values in the custom_openssl.cfg file.

Lines 76-78: Create a folder based on the Name-OUName combination to store all files used in certificate generation (CSR, private key, certificate, copy of the root CA chain certificates).

Lines 81-94: Customize the fields from the custom_openSSL.cfg file to match the specific server-service combination, according to the CSV file.

Lines 96-114:  We copy the template .cfg file to the working directory and replace each corresponding row with our values for name, city, state, etc. This is not written in the cleanest powershell for string manipulation, but it gets the job done.

Lines 116-131: This is where we generate the CSR to send to the CA. As per VMware’s instructions this is a 2 step process, first we generate the CSR, then we convert the private key to RSA format.

Lines 136-146: We use the windows tool certreq  to request a certificate and also retrieve the certificate. This is where being local admin in an elevated prompt comes into play, and also where automatic retrieval of certificates comes in handy.

Lines 151-159: We copy over the individual certificates of the Root/Intermediate/Issuing CAs to the working directory. Then we create a file called chain.pem, which will be the .PEM chained certificate file for the given server-component combination specified in the CSV.

Important!!! This is where reading the VMware KB wrong can cost you, and you won’t realize it until you run the SSL Automation Tool. I quote from the KB:

“Open the Root64.cer file in Notepad and paste the contents of the file into the chain.pem file right after the certificate section. Be sure that there is no whitespace in the file in between certificates.

Note: Complete this action for each intermediate certificate authority as well.”

This is where I initially made a mistake, and did it the wrong way.

  • I pasted the certificate in the .pem
  • I pasted the root after the certificate in the .pem
  • I pasted the intermediate/issuing CA certificates in the .pem after the root certificate

It is “do the same for each intermediate certificate” as in “sandwich your intermediate CAs certificates between the root and the certificate in descending order”, THE EXACT OPPOSITE of how Windows will display the certificate chain in the GUI. This is why I asked you in the first place to number your certificates in a specific way.

As a result, if you named your CA certificates like I explained above this code snippet will arrange them in the proper order and chain them in the .pem file correctly.

$chain_pem = "$rootdir\$($strHost.Name)-$($strHost.OUName)\chain.pem"
gc $crt | add-content $chain_pem</span>
$roots = dir "$rootdir\$($strHost.Name)-$($strHost.OUName)" root*.cer | sort name -desc
$roots | % {gc $_.FullName | add-content $chain_pem }

That wraps it up, needless to say, test this script before you let it run loose on your environment. Since the script has a lot of “Read-Host” in it, it is designed to run with pauses, to give you a chance to review the output, and cancel if there is a problem. After all you are running commands against your Certificate authority, so handle with care.

Once you have all the files you should follow step 2, and actually use the SSL automation tool.

Replace vCenter Certificates using the SSL Automation Tool

For reference here is VMware’s KB if you missed it above for using this tool. I don’t have much to say about this, other than stick to the document, but here are a few tips that will make your life easier:

  • If you have multiple servers running each service, it is best that you copy ALL the folders generated by the script (for each server-service combination) to each of the individual servers, and you can freely dispose of them once you successfully replaced the certificates.
  • Before you go ahead and copy over the Automation Tool, take the time to modify the SSL_Environment.bat file pointing each variable to its respective file/value. Then copy the SSL_Environment file to each server. This way each time you run the tool to update a certificate it will always know where to pick “inter-component-trust” certificates, user prompts and so on.
  • When you fill in the “set sso_admin_user=” variable put it in the format “user@domain” as it will give an invalid credentials error when you will run some steps.

I know it is a long read, but it is not an easy topic and I hope it helped you in your environment. Let me know in the comments if there is a way to improve this or an easier way to do this whole process.


Print pagePDF pageEmail page
Tagged , , , . Bookmark the permalink.

2 Responses to Automate Replacing of Certificates in vCenter 5.1

  1. Ytsejamer1 says:

    I’m a bit confused on steps 8…in the custom_openssl.cfg file, am I replacing the example values or leaving them in place to be substituted by either the tool or otherwise? You said copy and paste it exactly as is…so not quite sure if I should adjust values or not.

    subjectAltName = DNS:ServerShortName, IP:ServerIPAddress, DNS:server.contoso.com #examples only

    Same goes for step 9…what needs to be replaced?

    • Ionut Nica says:

      Hi,

      Step 8. The custom_openssl.cfg file should be a copy paste of the content I gave in the article. It is quite important the file be exactly as it is listed, as my script is replacing some lines from that file, and if the line numbers don’t match you get a busted file and probably a non working CSR or an incorrect one.
      In the case of the Subject Alternate Names, the complete SAN line will be replaced by the script, so you can leave it with “contoso.com”.
      Step 9 – This is the source CSV file, it needs to contain the short DNS name of the server where each vCenter component is (“Name” column header), the Qualified domain name (“Domain” column header), and the name of the components you wish to replace certificate for (i listed all, you can eliminate the ones you won’t be replacing certificates for).