====== VMware vCenter/ESXi CBT Bug Identification Check ======

Latest revision: 2020-10-08

===== The VMware CBT Bug =====

To enhance critical notification to our customers using our VMware vCenter/ESXi backup solution, a VMware version check feature on each backup job has been added, as there is a known VMware Changed Block Tracking (CBT) bug which exists on the following versions of VMware vCenter/ESXi:

  * VMware vCenter/ESXi v5.0
  * VMware vCenter/ESXi v5.1
  * VMware vCenter/ESXi v5.5
  * VMware vCenter/ESXi v6.0

This known VMware CBT bug can sometimes return incorrect changed sectors on a guest virtual machine during a VMware backup. As the CBT is used by the CloudBacko Pro for VDDK backup modes, it affects the integrity of both incremental and differential backups.

<WRAP info>
==== Information from VMware ====

  * For VMware vCenter/ESXi v5.0, v5.1, and v5.5, refer to [https://kb.vmware.com/s/article/2090639]
  * For VMware vCenter/ESXi v6.0, refer to [https://kb.vmware.com/s/article/2136854]
</WRAP>

If a VMware VDDK mode backup is performed on any of the affected versions (i.e. v5.0, v5.1, v5.5 and v6.0), even if the VMware vCenter/ESXi was subsequently upgraded to a later version, CloudBacko Pro will not be able to restore the guest VMs due to the known VMware CBT bug, even though all backup jobs are recorded as successful by the CloudBacko Pro. Furthermore, as it is a VMware bug, the CloudBacko Pro Periodic Data Integrity Check (PDIC) and Data Integrity Check (DIC) will not be able to detect this issue.

As part of the backup job, CloudBacko Pro v4.1.4 (or above) will check the current VMware version to identify if it is affected by the CBT bug. If the VMware version detected is related to the affected version (i.e. v5.0, v5.1, v5.5 and v6.0), then CloudBacko Pro will show a warning in the Backup Report indicating that the VMware host is affected by the CBT bug.

===== Bug Identification Check in v4.1.4 =====

Below is a sample of backup report for a VMware ESXi 5.1.0 host with the affected version:

{{public:features:cbt-bug-identification-check:cbt-bug-01a.png}}

The warning message will also indicate the VMware version or build no. which contains the fix for the CBT issue:

**VMware ESXi 5.1.0 build-1157734** is the current problematic version. The following patch needs to be applied: **FixedBuildNo: 2323236**. 

| Current backup source includes ESXi server(s) which could sometimes return incorrect changed sectors (KB2090639). Please upgrade the following ESXi server(s) to the fixed build to avoid backup incorrect backup data: [Name:localhost.localdomain (VMware ESXi 5.1.0 build-1157734), FixedBuildNo:2323236]. Reference: VMware ESXi 5.0.x, 5.1.x and 5.5.x (https://kb.vmware.com/s/article/2090639) and VMware ESXi 6.0.x (https://kb.vmware.com/s/article/2136854) | 

===== Resolution =====

To resolve this problem, it is strongly recommended to perform the following:

  - Apply the VMware patch or upgrade the VMware vCenter/ESXi.
  - Perform a full backup of the affected guest VMs.

<WRAP info>
==== Workaround for vCenter/ESXi v5.0 and v5.1 ====

Although CloudBacko v4 (or above) no longer supports the creation of new VMware backup set for VMware vCenter/ESXi v5.0 and v5.1, backup sets which are upgraded from the previous versions are still supported with the following workaround:

  - Download the VDDK v6 API files from here: https://download.ahsay.com/support/vddk6.zip
  - Stop the AhsayOBM services.
  - Stop the AhsayOBM NFS service.
  - Make a copy of the current folder %OBM_INSTALL%\bin\x64.
  - Using Windows Explorer open folder %OBM_INSTALL%\bin\x64.
  - Delete the following 13 .dll files from the %OBM_INSTALL%\bin\x64 folder.
    * diskLibPlugin.dll
    * libexpat.dll
    * lookup-types.dll
    * msvcp120.dll
    * msvcr120.dll
    * pcre.dll
    * sqlite3.dll
    * ssoclient.dll
    * vddkVimAccess.dll
    * vim-types.dll
    * vmacore.dll
    * vmomi.dll
    * zlib1.dll
  - Unzip the downloaded vddk.zip file and copy the contents %OBM_INSTALL%\bin\x64 folder.
  - Start the AhsayOBM services.
  - Start the AhsayOBM NFS service.
</WRAP>
<WRAP info>
==== List of VDDK 6 API files ====
    * glib-2.0.dll
    * gobject-2.0.dll
    * gthread-2.0.dll
    * gvmomi.dll
    * iconv.dll
    * intl.dll
    * libcurl.dll
    * libeay32.dll
    * liblber.dll
    * libldap.dll
    * libldap_r.dll
    * libxml2.dll
    * ssleay32.dll
    * VddkJNI.dll
    * vixDiskLib.dll
    * vixDiskLibVim.dll
    * vixMntapi.dll
</WRAP>