Corrupted image profiles in ESXi breaking HA agent installs and VIB installations

Ever wonder why your update just won’t install or why HA just won’t enable on a host? Well wonder no more!

We just upgraded to vSphere 6.5 Update 2 and had a host that would not install the HA agent. No amount of “Reconfigure for vSphere HA” or disabling/re-enabling HA would get the agent to install. There’s a KB article that talks about uninstalling the HA agent manually, so I gave that a try and lo and behold running esxcli software vib remove vmware-fdm came back with an error. Running esxcli software vib list showed only the vmware-tools VIB. This didn’t seem right, so I did some additional digging and found this KB article that worked out the problem (yes it says SDDC Manager, a function of VMware Cloud Foundation, but I saw this on a non-VCF host and it’s an ESXi issue, so I wouldn’t worry about titles).

Copy the imgdb.tgz file from a known good ESXi host with the same VIBs, upgrades and Patches.

To copy the imgdb.tgz file from a ESXi host:
 

1.       On the working ESXi host, copy a imgdb.tgz file by running this command:

cp /bootbank/imgdb.tgz /vmfs/volumes/shared-LUN
 

2.       On the corrupt host, move the imgdb.tgz file to /tmp by running this command:

cp /vmfs/volumes/shared_LUN/imgdb.tgz /tmp
cd /tmp
tar -xzf imgdb.tgz
 

3.       Copy profile files to the profile directory by running this command:

cp /tmp/var/db/esximg/profiles/* /var/db/esximg/profiles/
 

4.       Copy VIBs to the VIB repository by running this command:

cp /tmp/var/db/esximg/vibs/* /var/db/esximg/vibs/
 

5.       Remove the corrupt imgdb.tgz from the bootbank by running this command:

rm /bootbank/imgdb.tgz
 

6.       Move the correct imgdb.tgz file into the bootbank by running this command:

cp /tmp/imgdb.tgz /bootbank/
 

7.       Backup configuration changes by running this command:

/sbin/auto-backup.sh
 

8.       Restart the ESXi host.

The issue is that the image profile gets corrupted and blocks installs and uninstalls of VIBs. This includes HA as that installs a VIB as part of the enabling of the feature. I would also expect to possibly see this as part of an NSX host preparation failure.

The fix itself is pretty simple. Find a host with a similar configuration, copy over the image profile, unpack the files to the proper locations and reboot.

After the reboot, the image profile shows up and the VIB install (or in my case, vSphere HA enables) successfully.