Replacing vSphere 6.0 certificates using VMCA as a Subordinate CA

Replacing vSphere 6.0 certificates using VMCA as a Subordinate CA

In vSphere 6.0 the VMCA (VMware Certificate Authority) was introduced. This is basically vSphere’s own CA and it’s purpose is to simplify certificate generation and implementation in vSphere, in conjunction with VECS (VMware Endpoint Certificate Store)

While I do agree it does simplify the whole process, it’s not without its limitations and known issues. Hopefully this guide will help you avoid those pitfalls.

Firstly let me explain the small lab environment I will use.

·         I have a Root CA on my domain controller (dc.domain.com)

·         I have an Intermediate CA (interca.domain.com)

·         I have a Platform Services Controller 6.0 U2 Appliance (psc.domain.com)

·         I have a vCenter Server 6.0 U2 Appliance (vc.domain.com)

Create the Certificate Templates

In this guide we are using a Microsoft Certificate Authority.

Review https://kb.vmware.com/kb/2112009 and perform the steps outlines in the sections ‘Creating a new template for vSphere 6.0 to use for VMCA as a Subordinate CA’ and also ‘Adding a new template to certificate templates’

Generate VMCA Certificate Signing Request (CSR)

SSH to the Platform Services Controller (or vCenter Server if using VC w/ Embedded PSC)

Enable the BASH shell and set it to the default shell (we’ll need that when uploading the new certificate files)

shell.set --enabled True

shell

chsh -s /bin/bash root

Reference: https://kb.vmware.com/kb/2100508

1. Launch the certificate-manager utility from /usr/lib/vmware-vmca/bin/certificate-manager

/usr/lib/vmware-vmca/bin/certificate-manager

2. Select Option  2. Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates.

3. The certificate-manager will ask a question about generating all certificates using configuration files. We will answer Y to this question

4. Provide the administrator@vsphere.local credentials

5. Next we will be asked to configure the configuration file for MACHINE_SSL_CERT. The important bit here is to provide the FQDN for ‘Hostname‘ and I also recommended providing the FQDN for the ‘Name‘ field too.

6. Next we will be asked to configure the machine solution user configuration file. Same as the Machine_SSL_CERT in the previous step, provide the FQDN for the ‘Hostname‘ field. For the ‘Name‘ field use the format <solution_user>-fqdn. i.e. machine-psc.domain.com

Important: It is crucial that the subject information of all solution user certificates in this environment are kept unique.

7. Next we will be asked to configure the vsphere-webclient solution user configuration file. Repeat the same process as step 6 above. i.e. Make the ‘Name‘ field vsphere-webclient-psc.domain.com

8. If this was a vCenter Server with Embedded PSC there would be two more solution user configuration files to configure in the same manner; vpxd and vpxd-extension. Since this is an external PSC only two solution users exist.

9. Select Option 1: Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate

10. Provide a path to save the resulting csr and key file. In this example I just use the /tmp directory.

11. Next we will be asked to configure yet another configuration file called certool.cfg. This will control the information that our new VMCA certificate will contain. Again, provide the FQDN for the ‘Hostname‘ field and for the ‘Name‘ field use the format of VMCA-FQDN. i.e. VMCA-psc.domain.com

12. The certificate-manager will then generate the vmca_issued_csr.csr and vmca_issued_key.key in the location you specified earlier.

13. Using WinSCP or similar, copy the vmca_issued_csr.csr off the PSC Appliance.

14. Provide this csr to your certificate team to be issued with the new certificate. I will walk through this process using a Microsoft CA

Important: There is a known issue in 6.0 U2 where the CSR created for the VMCA as a Subordinate does not have the required attributes. Most Microsoft CA’s will force the inclusion of the attributes based on the certificate template used.

Check the certificate you just created from your CA by running the following OpenSSL Command:

openssl x509 -in vmca.cer -noout -text

The attributes we need present are CA:TRUE, Digital Signature, Certificate Sign, CRL Sign

 X509v3 Basic Constraints: critical

        CA:TRUE

 X509v3 Key Usage: critical

        Digital Signature, Certificate Sign, CRL Sign

If it is missing these attributes then you will need to create the CSR a different way by following this KB Article http://kb.vmware.com/kb/2145544

Obtain the signed Certificates from a Microsoft CA

1. Open a browser to your certificate authority web interface. i.e. http://vmwareca.domain.com/certsrv

2. Select ‘Request a certificate then select ‘advanced certificate request‘

3. Open the vmca_issued_csr.csr in a text editor like notepad and copy and paste the entire contents into the ‘Saved Request‘ text field and choose your VMCA Root certificate template from ‘Certificate Template‘ drop down and then hit ‘Submit‘

4. Select ‘Base 64 encoded‘ and then hit ‘Download certificate chain‘. You will be prompted to download and save a certnew.p7b file.

5. Open the certnew.p7b file in Windows and drill down to the Certificates container. You should see the new VMCA Root Certificates as well as your Root MS CA and any Intermediate MS CA certificates.

6. Export each of these certificates. In my lab I exported root.cer, inter.cer and vmca.cer

Right-Click > All Tasks > Export

Next > Base-64 encoded X.509 > File Path > Next > Finish

7. Open a Command Prompt to the location you exported the above certificates. Next we need to concatenate them into a single certificate chain, with the new VMCA on top, followed by the Intermediate CA, followed by the Root CA.

From the command prompt use the more command to concatenate. We will concatenate into a file called vmca_issued_cer.cer to keep with the naming convention of the files generated by the Certificate-Manager.

more vmca.cer >> vmca_issued_cer.cer

more inter.cer >> vmca_issued_cer.cer

more root.cer >> vmca_issued_cer.cer

8. Copy the vmca_issued_cer.cer up to /tmp/ on the PSC

Implement the signed Certificate as a new VMCA

1. Return to the Putty Session open to the PSC.

If you have closed this, return to the certificate-manager by running

/usr/lib/vmware-vmca/bin/certificate-manager

Select Option 2, then Option 1

2. Select Option 1: Continue to importing Custom certificate(s) and key(s) for VMCA Root Signing certificate

3. Provide the full path to the new VMCA Certificate Chain we created and then the Key that was created earlier

Please provide valid custom certificate for Root.

File : /tmp/vmca_issued_cer.cer

Please provide valid custom key for Root.

File : /tmp/vmca_issued_key.key

4. Ensure you have entered all the information correctly and also ensure you have taken a snapshot of the PSC and VC machines before answering ‘Y‘ to continue

You are going to replace Root Certificate with custom certificate and regenerate all other certificates

Continue operation : Option[Y/N] ? : Y

5. The Certificate-Manager will then replace the VMCA with your new Subordinate CA certificate and replace the certificates on the PSC (MACHINE_SSL_CERT, machine and vsphere-webclient solution user certificates)

It will also update the relevant service endpoints with the new MACHINE_SSL_CERT.

Lastly it will restart all the services on the PSC.

You are going to replace Root Certificate with custom certificate and regenerate all other certificates

Continue operation : Option[Y/N] ? : Y

Get site nameCompleted [Replacing Machine SSL Cert...]

cork

Lookup all services

Get service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b

Update service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b; spec: /tmp/svcspec_xVqq3K

Get service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc

// Snip //

Get service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health

Don't update service 79b06bf4-a18d-4f9e-b5a9-04affc35d2a4_com.vmware.vsan.health

Get service a82770ef-91ec-4e97-ba5a-b48fa3d5a371

Don't update service a82770ef-91ec-4e97-ba5a-b48fa3d5a371

Get service 7137e349-af59-4d0e-a03f-d235c9406740

Don't update service 7137e349-af59-4d0e-a03f-d235c9406740

Get service 3996a8c7-40a7-47e6-8912-bc4688dbbc59

Don't update service 3996a8c7-40a7-47e6-8912-bc4688dbbc59

Get service 29e3287a-f635-4b2f-aded-5241cbf25395

Don't update service 29e3287a-f635-4b2f-aded-5241cbf25395

Updated 9 service(s)

Status : 100% Completed [All tasks completed successfully]

Note: If this is a vCenter Server with Embedded PSC you would be complete and you would see more Updated Services.

If your vCenter Server is External then proceed to the next section.

Replace the Machine SSL Certificate on the vCenter Server from the new VMCA

1. Open a Putty session to the vCenter Server

2. Restart the vCenter Server service first before proceeding.

service-control --stop --all

service-control --start --all

3. Launch the Certificate-Manager

/usr/lib/vmware-vmca/bin/certificate-manager

4. Select Option 3 Replace Machine SSL certificate with VMCA Certificate

5. Provide the administrator@vsphere.local credentials

6. Since this is a vCenter Server with an external PSC you will be asked to provide the IP Address of the PSC.

Performing operation on distributed setup, Please provide valid Infrastructure Server IP.

Server : 192.168.2.101

7. Similar to before, we will be asked to configure the configuration file for MACHINE_SSL_CERT. The important bit here is to provide the FQDN for ‘Hostname‘ and I also recommended providing the FQDN for the ‘Name‘ field too.

8. Ensure the information entered is correct before answering ‘Y‘ to continue. Also ensure you have taken a new snapshot of the PSC and VC Machines.

You are going to regenerate Machine SSL cert using VMCA

Continue operation : Option[Y/N] ? : Y

9. The vCenter Server MACHINE_SSL_CERT will be replaced with a new one issued by the VMCA as a Subordinate, the relevant service endpoints with the new MACHINE_SSL_CERT and the services will be restarted.

You are going to regenerate Machine SSL cert using VMCA

Continue operation : Option[Y/N] ? : Y

Get site nameompleted [Replacing Machine SSL Cert...]

cork

Lookup all services

Get service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b

Don't update service cork:586c2bb6-9078-4bd7-8ba9-ddc411798c1b

Get service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc

Don't update service cork:fd3833cd-d96a-454e-9ccc-ea25d0befdfc

Get service cork:a46b5ede-2566-4d5a-9749-e83cbb644449

Don't update service cork:a46b5ede-2566-4d5a-9749-e83cbb644449

Get service be040cf2-5668-46fc-a013-3571745c7e12

Don't update service be040cf2-5668-46fc-a013-3571745c7e12

Get service f73219a5-730e-4429-81c2-4de727d16f73

// Snip //

Update service 08970f6b-2b84-4e84-81f6-04d87ba44e49_kv; spec: /tmp/svcspec_nIUxUp

Get service 4d30cfc1-d4eb-4303-8bfc-c2189cd1e7a1

Update service 4d30cfc1-d4eb-4303-8bfc-c2189cd1e7a1; spec: /tmp/svcspec_lkUrH5

Get service f371971c-4689-4f87-ae67-7db58f4d56df

Update service ac4d2f3f-2cd4-4a92-8ac4-8cadfb2ddb30; spec: /tmp/svcspec_Qej_nM

Update service a82770ef-91ec-4e97-ba5a-b48fa3d5a371; spec: /tmp/svcspec_M7zwgb

Get service 7137e349-af59-4d0e-a03f-d235c9406740

Update service 7137e349-af59-4d0e-a03f-d235c9406740; spec: /tmp/svcspec_BmOy3p

Get service 3996a8c7-40a7-47e6-8912-bc4688dbbc59

Update service 3996a8c7-40a7-47e6-8912-bc4688dbbc59; spec: /tmp/svcspec_b2H6M2

Get service 29e3287a-f635-4b2f-aded-5241cbf25395

Update service 29e3287a-f635-4b2f-aded-5241cbf25395; spec: /tmp/svcspec_OlFACZ

Updated 21 service(s)

Status : 100% Completed [All tasks completed successfully]

Replace the Solution User Certificates on the vCenter Server from the new VMCA

To be honest, replacing any solution user certificate is completely optional as they are not directly exposed to the end user. But if you require them to also be replaced by your new VMCA Subordinate then the following should be performed.

1. Launch the Certificate-Manager

/usr/lib/vmware-vmca/bin/certificate-manager

2. Select Option 6 Replace Solution user certificates with VMCA certificates

3. The certificate-manager will ask a question about generating all certificates using configuration files. We will answer Y to this question

4. Provide the administrator@vsphere.local credentials

5. Since this is a vCenter Server with an external PSC you will be asked to provide the IP Address of the PSC.

Performing operation on distributed setup, please provide valid Infrastructure Server IP.

Server: 192.168.2.101

6. Next we will be asked to configure the machine solution user configuration file. Provide the FQDN for the ‘Hostname’ field. For the ‘Name’ field use the format <solution_user>-fqdn. i.e. machine-vc.domain.com

Important: It is crucial that the subject information of all solution user certificates in this environment are kept unique.

Configure the vsphere-webclient, vpxd, vpxd-extension and solution user configuration files in the same manner. Ensuring that every solution user in the SSO Domain is unique.

If you follow my advice and make the ‘Name’ value <solution_user>-<fqdn> this will ensure that.

7. Ensure the information entered is correct before answering ‘Y‘ to continue. Also ensure you have taken a new snapshot of the PSC and VC Machines.

You are going to regenerate Solution User Certificates using VMCA

Continue operation: Option[Y/N] ? : Y

8. The machine, vsphere-webclient, vpxd and vpxd-extension solution user certificates will be replaced with new ones issued by the new VMCA and the services will be restarted on the vCenter Server.