Unable to SSH into the vCenter Server, fails with an error.

[Error]

/etc/profile.d/proxy.sh: line 11: Enable: command not found

Traceback (most recent call last):

File “/usr/lib/applmgmt/base/bin/vherdrunner”, line 8, in

vherdrunner.start(vherdrunner.directories)

File “/usr/lib/applmgmt/base/bin/vherdrunner.py”, line 130, in start

exec(code, childGlobals)

[Cause]

There was an extra p before the # sign wrong characters in /etc/sysconfig/proxy file.

less /etc/sysconfig/proxy

p# Enable a generation of the proxy settings to the profile.

[Resolution]

Removed the extra character p before the # sign and were able to ssh to vCenter.

less /etc/sysconfig/proxy

# Enable a generation of the proxy settings to the profile.

Note: – Review the /etc/sysconfig/proxy file for any extra or special characters & remove them (take a backup of the origina file).

Migrating a virtual machine between two different vDS version fails with an error.

What caused this error?

When attempting to migrate a virtual machine from one vSphere Distributed Switch (vDS) to another, you experience these symptoms:
The migration fails.

You see these errors in the vSphere Web Client similar to:

The target host doesn’t support the virtual machines current hardware requirements. The destination virtual switch version or type (VDS 7.0.0) is different than the minimum required version or type (VDS 6.6.0) necessary to migrate VM from source virtual switch.

Why do you see this?

This issue occurs because there are comparisons being made between the vDS on the source and destination for the vMotion operation. The vDS must match. Otherwise, this would mean the destination vDS is not compatible.

How to fix this?

This is an expected behavior when migrating between mixed vSphere Distributed Switches.

To resolve this issue, upgrade your vDS switch with the lower version to match that of the higher one in your infrastructure.

How do we workaround this without vDS upgrade?

  1. vCenter Server 6.5.x and vCenter Server 6.7.x
  2. Log in to the vCenter Server using the HTML5 or vSphere Web Client.
  3. Highlight your vCenter Server name in the left-hand column and then click on the Configure tab on the right.
  4. Go to Advanced Settings and click Edit Settings.
  5. At the bottom of the pop-up window, add the following property in the Name section:

config.migrate.test.NetworksCompatibleOption.AllowMismatchedDVSwitchConfig

  1. Set the value to true.
  2. Click Add.
  3. Click Save.
  4. Re-try the migration.

For vCenter Server 7.x and later

  1. Log in to the vCenter Server using the HTML5 or vSphere Web Client.
  2. Highlight your vCenter Server name in the left-hand column and then click on the Configure tab on the right.
  3. Go to Advanced Settings and click Edit Settings.
  4. At the bottom of the pop-up window, add the following property in the Name section:

config.vmprov.enableHybridMode

  1. Set the value to true.
  2. Click Add.
  3. Click Save.
  4. Re-try the migration.

Note: After enabling hybrid mode in vCenter, the target DVS version must be at least 6.0.0.

Performing a VASA Provider’s certificate replacement after vCenter Server convergence, would result in virtual volumes (vVOLs) going inaccessible from all ESXi host’s inventory.

What causes this?

1. The convergence workflow installs RPMs related to the PSC services which also means a new VMware Certificate Authority (VMCA)
instance is created on the embedded VC node.

2. VMCA creates a new VMCA root certificate which in turn is used for future certificate requests that the embedded node handles.

3. While the old certs are retained maintaining VC<-> host communication, other solutions like vVOl do not operate as the new certs provided to VASA providers have new ROOT certificte details whereas the hosts still have old ones causing vVol workflow to break.

How do you resolve this?

Renew or Refresh ESXi Certificates connected to vcenter server.

https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.security.doc/GUID-ECFD1A29-0534-4118-B762-967A113D5CAA.html

The certificate refresh has to be done manually per host.

Note:Bulk certificate management is currently not possible from vCenter Server UI at this time.

SSO Domain Re-point fails in vCenter 6.7 at Authz data export

This article about how to repoint Embedded PSC in one sso domain to another embedded domain in same of different sso domain.

Why do this?

  • To have the both vCenter’s connected under ELM(Enhanced Linked mode)
  • It will help in managing multiple vCenter’s with one user interface

How to do it?

We can run the re-point command in pre-check mode and execute mode.

Pre-check helps us validate the current environment and provide any potential errors we can encounter before we execute the command.

++Command Syntax :
cmsso-util domain-repoint -m pre-check –src-emb-admin Administrator –replication-partner-fqdn vcsa2.gss.local –replication-partner-admin Administrator –dest-domain-name vsphere.local

cmsso-util domain-repoint -m execute –src-emb-admin Administrator –replication-partner-fqdn vcsa2.gss.local –replication-partner-admin Administrator –dest-domain-name vsphere.local

  1. Pre-check mode in 6.7 u2 fails during the authz Data export

++In the /var/log/vmware/cloudvm/domain_consolidator.log you see the following error:

2019-04-25T20:49:29.215Z INFO domain_consolidator Started required services.
2019-04-25T20:49:29.659Z INFO domain_consolidator RC = 1
Stderr = Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
Exception in thread “main” java.lang.NoClassDefFoundError: org/springframework/context/support/AbstractApplicationContext
        at com.vmware.vim.vmomi.core.types.VmodlContext.initContext(VmodlContext.java:61)
        at com.vmware.vim.vmomi.core.types.VmodlContext.initContext(VmodlContext.java:42)

++Fix was to upgrade the vCenter server to vCenter 6.7 U3 version

++Workaround for the issue:

A. Validate the spring* files under /opt/vmware/lib64 should be with 4.3.20
B. Update of the spring version in vCenter 6.7 U2 from 4.3.9 to 4.3.20. The script /usr/lib/repoint/authzservice_component_script.py has hard set references to the 4.3.9 version.
You can run below command to edit all the entries in script
sed -i ‘s/4.3.9/4.3.20/g’ /usr/lib/repoint/authzservice_component_script.py

C.Proceed with running the pre-check command for successful completion

2. Pre-check mode failed in 6.7 U3 during Authz data export

++In the /var/log/vmware/cloudvm/domain_consolidator.log you see the following error:

2020-08-04T02:49:06.213Z INFO domain_consolidator Started required services.
2020-08-04T02:49:07.908Z INFO domain_consolidator RC = 1
Stderr = Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
Exception in thread “main” java.lang.Exception: QueryClient creation failed for VC:vcsa1.gss.local. Check ‘domain_data_export.log
at com.vmware.vim.dataservices.ExportAuthzData.main(ExportAuthzData.java:224)

2020-08-04T02:49:07.909Z INFO domain_consolidator Export of Authz Data failed. Exception {
“resolution”: null,
“problemId”: null,
“detail”: [
{
“id”: “install.ciscommon.command.errinvoke”,
“translatable”: “An error occurred while invoking external command : ‘%(0)s'”,
“args”: [
“Stderr: Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M\nException in thread \”main\” java.lang.Exception: QueryClient creation failed for VC:vcsa1.gss.local. Check ‘domain_data_export.log’\n\tat

++In domain_data_export.log we could see error mentioned below indicates STS certification validation failed.

2020-08-04T02:49:07.814Z [main DEBUG com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] Sending SOAP request to the STS server
2020-08-04T02:49:07.833Z [main DEBUG com.vmware.vim.sso.client.impl.ssl.StsSslTrustManager opId=] The SSL certificate of STS service cannot be verified against the list of client-trusted
certificates

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

++Fix would be update the STS service Certificate in MOB with MACHINE_SSL_CERT certificate.

A. Validate the certificate from vCenter mob for STS

B. Open the MOB, go to https://vCenter_IP/lookupservice/mob?moid=ServiceRegistration&method=List in a browser. login using administrator@vsphere.local account

C. In the filterCriteria text field, modify the value field to have only the tags <filterCriteria></filterCriteria> and click Invoke Method. This displays the ArrayOfLookupServiceRegistrationInfo objects

D. Search for sts/STS on the page. Find the value of the corresponding sslTrust field. The content of that field is the Base64 encoded string of the old certificate

E. Copy and paste the string in the ArrayofString field in the row of the sslTrust name (next to the ArrayOfString type), and save the string as a file named sts.cer.

F: Note the Thumbprint of certificate by opening it.

G. Run this command to export the new certificate to a file:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert –store MACHINE_SSL_CERT –alias __MACHINE_CERT –output /temp/new_sts.crt

H.The Thumprint of sts.cer and New_sts.crt do not match and its caused the validation for STS service fail.

I. Command to update correct certificate information under mob.

python /usr/lib/vmidentity/tools/scripts/ls_update_certs.py –url https://psc.domain.com/lookupservice/sdk –fingerprint Thumbprint_of_sts.cer –certfile /temp/new_sts.crt –user administrator@vsphere.local –password Password


++ Run the command to perform Domain re-point in pre-check mode.