Recently I have been implementing a vSphere 6 design for a customer that is using the hybrid certificate implementation (Machine SSL) across multiple vCenter servers and Platform Services Controllers. This VMware blog here describes the general concept behind this
Custom certificate on the outside, VMware CA (VMCA) on the inside – Replacing vCenter 6.0’s SSL Certificate
One of the design decisions behind this choice of a hybrid deployment was the customer relies on a third party managed PKI service, and do not wish to change standard operating procedures to issue a subordinate CA template which is required to provision the VMCA as a subordinate CA.
The third party managed PKI requires the full certificate chain including an Intermediate and Root CA to be installed along with the web (Machine) certificate on all platform services controllers and vCenter Servers.
Whilst you can use and it is recommended to use the VMware Certificate Manager Utility to do this, I recommend researching the two KB articles and workarounds before you attempt this process to see if it applies to your environment
Using the Certificate Manager Utility in vSphere 6.0 does not utilize the Certool.cfg for CSR generation (2129706)
Replacing certificates using VMware vSphere 6.0 Certificate Manager fails at 0% with the error: Operation failed, performing automatic rollback (2111571)
These are the steps I used to implement CA signed Machine SSL certificates. This is using Windows based vCenter servers but the process is similar if you are using the vSphere appliance.
Start the certificate replacement first on your platform services controller if it is external to vCenter. If it is embedded, the process is the same
Step 1. Create a CSR Configuration File
Browse to C:\Program Files\VMware\vCenter Server\vmcad\certool.cfg and edit as follows according to your environment (back it up first)
Country = country code
Name = FQDN or short name of relevant vCenter or PSC
Organization = Organization name
OrgUnit = Organizational unit
State = state
Locality = your city
Email = Email address (optional)
Hostname = FQDN of vCenter or PSC
Step 2. Generate a CSR
Browse to C:\Program Files\VMware\vCenter Server\vmcad and run the following certool command to generate a CSR
certool.exe --initcsr --privkey=priv.key --pubkey=pub.key --csrfile=csr.csr --config=certool.cfg
The CSR, public key and private key will be exported to
C:\Program Files\VMware\vCenter Server\vmcad
You can check your generated CSR at https://cryptoreport.websecurity.symantec.com/checker/views/csrCheck.jsp to see if it is valid and formatted correctly before submitting to your CA.
Step 3. Submit your CSR to your relevant certificate authority.
Submit your certificate request to your certificate authority. Once you get your signed certificate back, make sure you also grab a copy of the intermediate and root CA along with the machine certificate. The certificates must be in PEM format (CER, PEM, or CRT) using Base64 encoding.
If you are missing a copy of the intermediate or root CA’s you can automatically generate a chain from your machine certificate at https://whatsmychaincert.com/
Make sure you tick the box “Include Root Certificate”
Step 4. Join the Machine, Intermediate and Root Certificates
Once you have a copy of all the certificates, open them up in Notepad and join together in the following order.
- Machine Certificate
- Intermediate Certificate
- Root Certificate
Make sure no white space exists in the certs and they are in the correct order, otherwise the certificate replacement will fail. Save a copy as fullchain.cer
The chained certificate should look as follows.
Step 5. Replace the the Machine SSL Certificates with chained CA certificates using vecs-cli
Next, you can replace the certificates with two methods, Manually using vecs-cli or using VMware’s certificate management script . I prefer using vecs-cli myself.
This method is supported and documented in the vSphere 6.0 Security Documentation here
Replace Machine SSL Certificates With Custom Certificates
If you are running an external platform services controller, remember to replace its certificate first before any vCenter Servers.
First, copy both the previously created fullchain.cer and the priv.key that was created from step two, to C:\Program Files\VMware\vCenter Server\vmafdd
Next, Stop all services, and then start only the services that handle certificate creation, propagation, and storage from an administrative command prompt.
For Windows this is run using service-control from an administrative command prompt at
C:\Program Files\VMware\vCenter Server\bin
service-control --stop --all
service-control --start VMWareAfdService
service-control --start VMWareDirectoryService
service-control --start VMWareCertificateService
The commands are case sensitive so you need to spell VMware incorrectly with a large W
Next, Switch to C:\Program Files\VMware\vCenter Server\vmafdd and run the following to delete the old Machine SSL Certificate entry in VECS..
vecs-cli entry delete --store MACHINE_SSL_CERT --alias __MACHINE_CERT
Then, add your custom chained certificate to the store along with the private key you generated previously in Step Two.
vecs-cli entry create --store MACHINE_SSL_CERT --alias __MACHINE_CERT --cert fullchain.cer --key priv.key
Not that the __MACHINE_CERT alias has two underscores. If you type it incorrectly, it will add multiple entries to VECS
You can check the certificate was added correctly to the store by running
vecs-cli entry list --store MACHINE_SSL_CERT -–text
Finally, start the services again by running service-control –start –all
After the services have started, browse to your vSphere webclient GUI and check the certificate. It should include the full certificate chain like the below screenshot
If you would prefer to use the VMware Certificate Manager rather than vecs-cli, for replacing the Machine SSL Certificate, refer to the following KB article.
Replacing a vSphere 6.0 Machine SSL certificate with a Custom Certificate Authority Signed Certificate (2112277)
You should provide the full certificate chain on the step ‘Please provide valid custom certificate for Machine SSL’. If you don’t, you won’t get an intermediate cert on the machine certificate that is provided to clients.
If you get issues with the vSphere Thick Client taking a long time to connect after SSL replacement, check that the certificate request included a subjectaltname that matches the hostname of the vCenter server. This is only a issue if your Certificate Common Name is different to the hostname. This problem does not affect the webclient.