VMware Troubleshooting

From SEPsesam
Jump to: navigation, search

Copyright © SEP AG 1999-2019. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Docs latest icon.png Welcome to the latest SEP sesam documentation version 4.4.3/4.4.3 Beefalo. For previous documentation version(s), check documentation archive.


Contents

VMware vStorage API

SEP sesam version > 4.4.3.22

Backup

VMware slow backup performance via NBD. OR: Backups are failing in SAN transport mode with VMs residing on VMFS 6.0

Problem

  • Slow backup performance via NBD. Another problem is that backups might fail in SAN transport mode.

Cause

  • vSphere 6.5 and 6.7 may have slow backup performance via NBD with VDDK 6.0.x due to the VMware policy concerning backward and forward compatibility for VDDK to support N-2 and N+1 releases.
  • With new vSphere 6.5 neither of VDDK 6.0.x versions provides SAN transport support when using datastores formatted with new VMFS 6.0 filesystem. Consequently, the backup will fail.

Solution

  • On Windows, VDDK is installed automatically during the SEP sesam installation. However, if you use a new vSphere version you should check the VDDK Compatibility Matrix to find its corresponding VDDK version and install it manually, if required. On Linux, you have to install the required VDDK version manually. Note that every new release of vSphere has a corresponding VDDK version; typically, the version number of VDDK matches the version number of vSphere. For details on the required version, see VDDK Compatibility Matrix.
VMware backup fails with error due to unrecognized extended LUN connected to a Linux data mover

Problem

  • VMware backup fails with error, such as: Error while reading data. Error: VixDiskLib: VixDiskLib_Read: Read 2048 sectors at 0 failed. Error 16000 (One of the parameters supplied is invalid) (DiskLib error 4096062: One of the parameters supplied is invalid) at 5024.

Cause

  • On the Linux data mover in VMware environments, when you extend LUN while online (without restarting your Linux system), the extended LUN will not be instantly visible from the Linux operating system. The extended disk is automatically adjusted on Windows, while on Linux you need to rescan SCSI bus manually. Consequently, the backup will fail.

Solution

  • To rescan SCSI bus on Linux without restarting it, use the following command when adding a new disc (X is the number of SCSI host to scan):
echo  “- – -” > /sys/class/scsi_host/hostX/scan
  • Use the following command when expanding an existing disc:
echo 1 > /sys/class/scsi_device/device/rescan

Single file restore

Single file restore is not working with VMware 6.0

Problem

  • When restoring single files on VMware 6.0, the restore fails with: Cannot open Thin/TBZ disks in a multiwriter mode. This log is also shown in the vCenter events.

Cause

  • This error appears if the SCSI bus sharing type on the proxy machine is set to Physical.

Solution

  • Shut down the proxy machine and change the type of SCSI bus sharing to None, so the virtual disks cannot be shared by other virtual machines.
When restoring a single file, SEP sesam cannot access storage on a VM that is configured with the SCSI LSI Logic SAS adapter

Problem

  • In SEP sesam version ≥ 4.4.3.48 in combination with a Linux proxy VM, when trying to select the restore source in the restore wizard (step 4: Select Files), no items are displayed in the browser.

Cause

  • This issue seems to be related to the SCSI controller of the proxy VM. When trying to restore a single file from a virtual machine, SEP sesam cannot access storage on a virtual machine that is configured with the SCSI LSI Logic SAS adapter.

Solution

  • Open the properties of the proxy VM in your vCenter and change the controller on the VM from the LSI Logic SAS controller (on some VMs, this is selected by default) to an LSI Logic Parallel controller. For more information, see VMware Docs article Change the SCSI Controller Type in the vSphere Client.
  • Restart the restore wizard. If there are still no items displayed in the restore browser, reboot the proxy VM and then click the Update button.
After updating an existing SEP sesam structure with a new VDDK version, it is no longer possible to mount a VMDK

Problem

  • After VDDK ≥ 6.5.x has been manually installed on Windows, it is no longer possible to mount a VMDK saveset on this host.

Cause

  • If another VDDK version, for example VDDK ≥ 6.5.x, is manually installed on Windows, the host requires a reboot after the update if you want to mount a VMDK on this host.

Solution

  • Reboot the host after the VDDK update, then run the following script from the command line:
  • vstor2install.bat in directory C:\Program Files\VMware\VMware Virtual Disk Development Kit\bin

SEP sesam version 4.4.3.22 and previous

Single file restore

Single item restore of migrated save sets does not work with version 4.4.3.22

Problem

  • Single item restore of migrated save sets does not work with version 4.4.3.22.

Workaround

  • This issue is solved with the patch. Contact support or Upgrade to a higher 4.4.3.x version.

SEP sesam version 4.4.2.87 and previous

Installation and configuration

SSL handshake error during browse of vSphere server

Problem

  • The name "VMware vSphere:" appears again after expanding it.
  • A pop up opens, which refers to this page.

Possible causes

  • This issue is related to VMware vSphere 5.5 and IBM Java due to changes in OpenSSL.

Solution

"Execution error of set_vddk.ps1 according vmware-vdiskmanager.exe"

Problem

  • The required msvcr90.dll for this application is not available.

Solution

Error: Load VDDK library failed: Cannot load: vixDiskLib.dll

Problem

  • You receive the warning Error: Load VDDK library failed: Cannot load: vixDiskLib.dll.

Possible causes

  • VDDK is only installed as a 32-bit version.

Solution

VM creation breaks with an error

Problem

  • VM creation breaks with the error:
svp-3980: Info:    Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf'
svp-1901: Error:   VM Exception: [VM creation failed  (error)].

Possible causes

  • Wrong video RAM size. Must be set to 4MB in auto-detect mode. The error message in host.d is:
opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Video Ram size edit not supported when auto-detect is True ./var/log/hostd.log:2013-04-23T14:43:56.923Z [61B7EB90 info 'vm:/vmfs/volumes/5087d83e-bbbbd91c-4218-001999da71cf/testEX-SERVER/testEX-SERVER.vmx' 
opID=49F494FA-000001D6-6a-7b-3e-ec-f4] Reconfigure failed: vmodl.fault.NotSupported

Solution

  • Analyse the VMware logs:
    • Use vCenter <Administration>, <export system logs> to select the affected ESX-Server as source. Choose <Logs> only from system logs and specify the desired export location.
    • Once the export is complete, unzip the tgz file from the export directory then locate the host.d file and search for errors.
Cannot display VMX file

Problem

  • Error occurs during download of VMX file.

Possible causes

Solution

  • Update your version of vCenter. This problem is not related to SEP sesam but to vCenter only.

Backup

Backup of VMs using VDDK 5.1.1 on Linux occasionally fails

Problem

Workaround

Backup of VMs with two VMDK files with the same file name does not work

Problem

  • If a virtual machine (named vmware) uses the VMDK files with the same name, e.g., [Datastore1]/vmware/vmware.vmdk and [Datastore1]/vmware_org/vmware.vmdk , the backup will fail.

Workaround

  • Rename one of the virtual disks to a name different than the first one.
"Error: VM Exception: [HostCommunication]."

Problem

  • A VADP backup ends with the error message "Error: VM Exception: [HostCommunication].".
  • In the event log of the hyper visor, the following error message occurs at the same time if
    • a snapshot should be created or deleted on this host
    • a virtual machine should be migrated to or from this host
    • a virtual machine on this host should be cloned
An error occurred while communicating with the remote host.

Possible causes

  • One of the management services on the hypervisor is not or not correctly running.

Solution

  • Log onto the ESXi server via ssh
  • Restart the vpxa Service
/etc/init.d/vpxa restart

See also: Restarting the Management agents on an ESXi or ESX host

VIX_E_FAIL

Problem

  • An error occurs during thr backup of the second VMDK.

Possible causes

  • Timeout in vCenter connection.

Solution

  • Update the VDDK library to version 1.2.1.
VIX_E_FILE

Problem

  • You receive one of the following warnings:
    • VIX_E_FILE_NOT_FOUND:
    • VIX_E_FILE_ACCESS_ERROR:
    • You do not have access rights to this file:
    • Thin/TBZ/Sparse disks cannot be opened in multi writer mode.

Possible causes

  • Wrong VMDK file has been linked for backup.

Solution

  • Start backup with the option -a qui=0. The system will boot in the normal state and show a message regarding safe boot. The screenshot below shows how to add this option:

Qui.jpg

Backup fails with error on a 64 Bit system even though VDDK is already installed

Problem

  • You receive the warning Backup fails with error on a 64 Bit system even though VDDK is already installed.

Possible causes

  • VDDK is only installed as a 32-bit version.

Solution

Backup stops with sbc-1500 error message

Problem

  • Backup stops with the message: sbc-1500: Error: VixDiskLib_Open() failed: [sesam] sesam_vm/sesam_vm.vmdk:Cannot connect to the host.

Possible causes

  • The file vixDiskLib.dll does not exist in the directory <SESAM_ROOT>\bin\sesam of the VMware data mover.

Solution

  • Copy the file C:\Program Files\VMware\VMware Virtual Disk Development Kit\vixDiskLib.dll to <SESAM_ROOT>\bin\sesam of your VMware data mover.
Encrypted backup failure

Problem

  • Encrypted backup stops with the warning: "Encryption/decryption operation failed".

Possible causes

  • Backup was done with bf64 (blowfish 64) encryption.

Solution

  • The encryption must be changed from bf64 to aes256.
Error: Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so

Problem

  • A VADP backup ends with: "Error: VM Exception: [Exit code from sbc: [1] - warning]”
  • In the log file you see the warning:
Could not load default plugins from /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so.

Possible causes

  • The incorrect LD_LIBRARY_PATH has been set.

Solution

  • Edit /etc/sesam2000.ini and add/modify the LD_LIBRARY_PATH line:

For 32-bit Linux:

  VERSION=4.2.1.41
  SM_BIN_SESAM=/opt/sesam/bin/sesam/
  ...
  LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib32:/usr/lib/vmware-vix-disklib/plugins32:$LD_LIBRARY_PATH

For 64-bit Linux:

  VERSION=4.2.1.41
  SM_BIN_SESAM=/opt/sesam/bin/sesam/
  ...
  LD_LIBRARY_PATH=/usr/lib/vmware-vix-disklib/lib64:/usr/lib/vmware-vix-disklib/plugins64:$LD_LIBRARY_PATH
  • If the problem persists after modifying the sm.ini and restarting the SEP sesam service, check if the libexpat.so.0 actually exists on the selected system:
vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so
       linux-vdso.so.1 =>  (0x00007fff539ff000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdab6937000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007fdab662d000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007fdab6428000)
       libcrypto.so.0.9.8 => /usr/lib64/libcrypto.so.0.9.8 (0x00007fdab6089000)
       libssl.so.0.9.8 => /usr/lib64/libssl.so.0.9.8 (0x00007fdab5e35000)
        libexpat.so.0 => not found
        
       libz.so.1 => /lib64/libz.so.1 (0x00007fdab5c1e000)
       libtypes.so => not found
       libvmomi.so => not found
       libvmacore.so => not found
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fdab5a07000)
       libc.so.6 => /lib64/libc.so.6 (0x00007fdab5692000)
       /lib64/ld-linux-x86-64.so.2 (0x00007fdab70b0000)
       libm.so.6 => /lib64/libm.so.6 (0x00007fdab5419000)
  • Search for the file libexpat.so.0:
       vcenter:/usr/lib64 # ls -lart libexp*
        -rwxr-xr-x 1 root root 196944 May 29  2009 libexpect5.44.1.11.so
        -rwxr-xr-x 1 root root 170984 Feb 20  2010 libexpat.so.0.5.0

Result: The file libexpat.so.0 does not exist

  • Resolve the issue:
ln -s libexpat.so.0.5.0 libexpat.so.0
  • First control after generating the soft link:
vcenter:/usr/lib64 # ls -lart libexp*
-rwxr-xr-x 1 root root 196944 May 29  2009 libexpect5.44.1.11.so
-rwxr-xr-x 1 root root 170984 Feb 20  2010 libexpat.so.0.5.0
lrwxrwxrwx 1 root root     17 Jul  2  2012 libexpat.so.0 -> libexpat.so.0.5.0
  • Second control after generating the soft link:
       vcenter:/usr/lib64 # ldd /usr/lib/vmware-vix-disklib/lib64/libdiskLibPlugin.so
       linux-vdso.so.1 =>  (0x00007fff90f39000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f83497de000)
       libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f83494d3000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007f83492cf000)
       libcrypto.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libcrypto.so.0.9.8 (0x00007f8349053000)
       libssl.so.0.9.8 => /usr/lib/vmware-vix-disklib/lib64/libssl.so.0.9.8 (0x00007f8348f02000)
 libexpat.so.0 => /usr/lib64/libexpat.so.0 (0x00007f8348cd8000) 
       libz.so.1 => /lib64/libz.so.1 (0x00007f8348ac2000)
       libtypes.so => /usr/lib/vmware-vix-disklib/lib64/libtypes.so (0x00007f834401a000)
       libvmomi.so => /usr/lib/vmware-vix-disklib/lib64/libvmomi.so (0x00007f83438c6000)
       libvmacore.so => /usr/lib/vmware-vix-disklib/lib64/libvmacore.so (0x00007f8342e34000)
       libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f8342c1d000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f83428a9000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f8349f88000)
       libm.so.6 => /lib64/libm.so.6 (0x00007f8342630000)
       libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f83423f4000)

Restore

Restoring VMware vSphere 6 does not work with SEP sesam 4.4.1

Problem

  • Backup of vSphere 6 works with capabilities of VDDK 5.5. But restore does not work at all with 4.4.1 due to check of vSphere version of SEP sesam GUI.

Workaround

  • No workaround exists for version 4.4.1. Upgrade to SEP sesam version ≥ 4.4.2 to be able to restore vSphere 6.
VM configuration restore uses a lot of hard disk space

Problem

  • If only the configuration of a VM containing thick provisioned VMDKs is restored, the new VMDKs will have the same size as the original VMDKs. This is because SEP sesam creates empty VMDKs during the restore and uses the full disk size for thick provisioned disks. (For thin provisioned disks 0 bytes are reserved.)

Workaround

  • Make sure that there is enough space in the target data store for restoring thick provisioned VMDKs related data.
VM Exception: [InvalidRequest]

Problem

  • During VM creation with OVF config file, you receive the warning: VM Exception: [InvalidRequest].

Possible causes

  • The OVF file used to create VM during restore is invalid. For example, the virtual CD-ROM with the ISO file is no longer available.

Solution

  • Restore only VM config files.

New Restore Task VM config.jpg

  • Remove the virtual CD-ROM with the ISO from the OVF file.
  • The file can be found in the SEP sesam directory: <SESAM_ROOT>/var/tmp/_vadp_/<vm name>/<saveset of backup>
CD-ROM specification part of ovf file
    <Item>
      <rasd:AddressOnParent>0</rasd:AddressOnParent>
      <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
      <rasd:ElementName>CD/DVD Drive 1</rasd:ElementName>
      <rasd:HostResource>ovf:/file/file2</rasd:HostResource>
      <rasd:InstanceID>7</rasd:InstanceID>
      <rasd:Parent>4</rasd:Parent>
      <rasd:ResourceType>15</rasd:ResourceType>
    </Item>
  • Start the same restore task and restore VM with VMDK

New Restore Task VM VMDK data.jpg

  • Edit the OVF file of the VM and remove all virtual devices that prevent the creation of the VM.
 svp-3980: Info:    Create VM from OVF file: 'C:\ProgramData\SEPsesam\var\tmp\_vadp_\EX-SERVER\SC20130423113643791@70RYUkBmDGz\EX-SERVER.ovf'
 svp-1901: Error:   VM Exception: [VM creation failed  (error)].
Restore of VMDK with transport mode SAN fails

Problem

  • Error occurs during restore shortly before all data are written. You receive the warning: "Restore of VMDK with transport mode SAN fails with "Write to disk failed: Error 1365799600129 at 4110" or similar message".

Possible causes

  • This issue is related to VMware, not to SEP sesam. Ordinarily, this problem only occurs if a VMDK has been created with a capacity value of MB (megabyte) instead of GB (gigabyte) or TB (terabyte) and in this case only if the value of the size is not a integer number (i.e. the value 16.3 MB is not a integer number, but 16 MB is).

Solution

  • VMware fixed the problem in version 5.5 of ESXi. If it is not possible to upgrade the ESXi, the restore can be done via all other transport modes (NBD,NBDSSL,HOTADD).
Restore of VMware machine via SAN with Windows data mover fails

Problem

  • Restore fails with the following message immediately after starting data transfer: Restore of VMware machine via SAN with Windows data mover fails with "SAN transport error: I/O - Operation failed".
2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.260+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19)
2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.262+02:00 [02192 error 'Default'] Incomplete write to, Wanted 65536 Got 0, Error 2 (19)
2014-10-13 13:32:45: sbc-3925: Info:    VDDK:    INFO: 2014-10-13T13:32:45.264+02:00 [11876 error 'Default'] San transport error: I/O Operation failed.
2014-10-13 13:32:45: sbc-1031: Error:   Cannot write to remote archive: [VixDiskLib: VixDiskLib_Write: Write to disk failed: Error 17592452332404352 at 4116.

Possible causes

  • The SAN volume on Windows data mover is offline, which implies read only access to this volume.

Solution

  • Change the disk status from offline to online by using the Windows disk management tool. Right-click the affected disk and change its status then try the restore again.
  • If the former is not possible, a transport mode other than SAN must be used. Possible are HOTADD, NBD or NBDSSL. The easiest way to do this is to enter -a trans=hotadd:nbd:nbdssl into the restore options of the backup task. This will be applied automatically during the restore.

GUI

GUI reports Login denied during browsing the vSphere farm

Problem

  • When vCenter user and password have been verified and the error is still running, you can check the VMware farm access without a graphical user interface (GUI).

Possible causes

  • Occasionally, there are problems with special characters in passwords. For a better analysis of these problems, it would help to check the VMware vSphere farm connection using the command line.

Solution

  • In the following command you have to replace the vCenter server name, username and password with your own environment's variables.

Pattern:

sbc_vadp -D -a username=<vCenter username>,password=<password>,url=https://<vCenter Hostname>/sdk,ignorecert=ignorecert "/VMware vSphere:"

For example:

sbc_vadp -D -a username=Administrator,password=mypassword,url=https://ws2008x64/sdk,ignorecert=ignorecert "/VMware vSphere:"

It should then be listed in the configured VMware Data Center. If that is not the case, then set the password without special characters. If the access test to the farm is successful, then the original password contained a character that caused the problem accessing the VMware farm.

CBT

After upgrading VMware vSphere the entire size of VMDK files may be backed up

Problem

  • After upgrading VMware vSphere the entire size of VMDK files may be backed up.

Solution

Error: At least one snapshot exists before CBT was enabled, which is not possible

Problem

  • You receive the warning: "Error: At least one snapshot exists before CBT was enabled, which is not possible"

Possible causes

  • VM is suspended.
  • Manual snapshots available.
  • Restart or shut down VM.
  • Delete manually created snapshots and then perform a reset CBT.
svp-1901: Error: VM Exception: [Exception=FileFault,dynamictype=null]. -> CBT-RESET needed

Problem

  • The exception error occurs when a snapshot is not deleted after a CBT backup.

Possible causes

  • CBT is not in a clean state.

Solution

  • Reset CBT as described below.
Resetting CBT via CLI or GUI

Soft reset (online via CLI)

Depending on SEP sesam version, use a relevant command for soft reset via CLI without shutting down the VM:

  • From SEP sesam v. 4.2.2.26 to v. 4.4.1.x, you have to execute the command in the directory <SESAM_ROOT>\bin\sesam.
  • Example

    sbc_vadp -b -a "action=softreset,server=ws2008x64,username=Administrator,password=secret" 
    "DC/linuxdbserver"
  • As of SEP sesam v. ≥ 4.4.1.x, use this command for soft reset.
  • Example

    sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)"
    

Soft reset (online via GUI)

As of SEP sesam version ≥ 4.4.1.x, you can reset CBT via GUI; note that this reset is task-related and can be done per each individual task. The Reset CBT button is available in the first tab of a backup task properties: Main Selection -> Tasks -> by Clients -> double-click the relevant VMware task to open its properties. Click Reset CBT option.

Hard reset (offline via CLI)

Depending on SEP sesam version, use a relevant command for hard reset via CLI:

  • For SEP sesam v. < 4.4.1.x, if a soft reset does not work or it is not available, use the following command for hard reset.
  • Example

    sbc_vadp -b -a "action=resetcbt,server=ws2008x64,username=Administrator,password=secret" 
    "DC/linuxdbserver"
  • As of SEP sesam v. 4.4.1.x, use this command for hard reset.
  • Example

    sm_cmd resetcbt -S qsbox1 -d "SEP Cloud" -V "cosinus (SEP)" -m hard
    

    The virtual machine must be powered off for this action.

CBT reset via the vSphere client

  • Alternatively, a CBT-reset can be performed via the vSphere client:

Online

  • The virtual machine does not need to be shut down to reset CBT:
  1. Open the properties of the affected virtual machine from Tasks > By Clients.
  2. Uncheck Enable Change Block Tracking (CBT) and save the task.
  3. Create a snapshot via vSphere Client of the affected virtual machine (none of the ticks needs to be selected).
  4. Delete the snapshot that was created before.
  5. In SEP sesam, activate the CBT option in the VM client settings and start the VM backup.

Offline

  • This step needs to be performed if a reset of CBT in online mode does not work. The virtual machine must be turned off for this step:
  1. Shut down the VM.
  2. Open the VMware Snapshot-Manager and delete all snapshots.
  3. Right-click VM -> Edit settings -> Options tab -> General -> Configuration Parameters.
  4. Set the value "ctkEnabled" to false.
  5. Set the value "scsi0:0.ctkEnabled" to false (NOTE: set the value to false for each disk).
  6. Open the folder where the VM *.VMDK files exists and delete all *-CTK.VMDK files
  7. Start the VM.
  8. Shut down the VM (this is required for the CTK table update).
  9. Start the VM again.
  10. In SEP sesam, activate the CBT option in the VM client settings and start the VM backup.

See also

VMware