Troubleshooting Guide

From SEPsesam
Revision as of 15:04, 28 March 2022 by Sta (talk | contribs) (Minor: changed link to Lotus troubleshooting/en (prepared for translation))
Other languages:

Template:Copyright SEP AG en

Welcome to the latest SEP sesam documentation version 4.4.3 Beefalo/5.0.0 Jaglion. For previous documentation version(s), check documentation archive.


Introduction

Additional resources
See also: Error Messages Guide

FAQ

Check FAQ to find the answers to most common questions.

This guide is intended to help you quickly identify and resolve problems and errors during setup, installation and during normal operation of your SEP sesam system. SEP sesam often serves as an indicator that there has been a change or event that impacts overall system performance. Changes in SEP sesam backup performance are often caused by system changes or failures.

Template:First steps/en Interpreting Error Messages/en Setting Log Level/en For details on how to set a log level for specific backup or restore task in the GUI or globally for SEP sesam kernel modules, see Setting Log Level. Installation and Configuration Troubleshooting/en Troubleshooting Authentication/en Backup Troubleshooting/en Disaster Recovery Troubleshooting/en GUI Troubleshooting/en Network Troubleshooting/en MS SQL Troubleshooting/en MS Exchange Troubleshooting/en NetWare Troubleshooting/en Micro Focus GroupWise Troubleshooting/en Oracle Troubleshooting/en Informix Troubleshooting/en Lotus Domino Server Troubleshooting/en VMware Troubleshooting/en

Citrix XenServer

Error when using quiesced snapshots

Problem

  • Using VM.snapshot_with_quiesce method fails in Citrix Hypervisor ≥ 8.1 or in the version ≥ 9.0.x.x drivers.

Cause

  • VSS and quiesced snapshots of Windows VMs capability have been removed in Citrix Hypervisor ≥ 8.1 and in the version ≥ 9.0.x.x drivers. They are only supported in Citrix Hypervisor ≤ 8.0.

⇒ Solution

  • If you want to continue using the quiesced snapshot feature with Windows VMs hosted on Citrix Hypervisor 8.0 and earlier, do not update to the 9.0.x.x drivers.

CBT backup fails because Hypervisor TLS certificate contains wrong IP address

Problem

  • CBT backup fails with the NBD client connection error:
Failed to get NBD client for device '******: NBD client connection error: hostname '****' doesn't match u'******

Cause

  • The TLS certificate of a Xen Server contains the wrong IP address. Therefore no NBD session can be established.

⇒ Solution

  • Check all Citrix Hypervisors in the Pool whether the IP address in the certificate matches.
  • On the Citrix Hypervisor where it does not match, execute the following commands:
  1. Create a backup of the existing certificate by using
    2. mv /etc/xensource/xapi-ssl.pem /etc/xensource/xapi-ssl.pem.$(date -u +%Y%m%d)
  2. Create a new certificate
    3. /opt/xensource/libexec/generate_ssl_cert /etc/xensource/xapi-ssl.pem <ip-address or FQDN>
  3. Restart XenAPI service
    4. systemctl restart xapi
  4. Restart NBD Service
    5. systemctl restart xapi-nbd

Message "Upload to URL ... failed" appears during restore, "insufficient space" error occurs during backup

Problems

  • Restore fails with an error:
"Upload to URL ... failed."

SR_BACKEND_FAILURE_44: : There is insufficient space to create snapshot on Storage Repository.

Possible cause

  • There is not enough space available on the default storage.
  • If the "insufficient space" error occurs during the backup, it might be related to the snapshot process. XenServer was unable to create a snapshot for the VM because there was not enough space in your Storage Repository.

⇒ Solution

  • Use XenCenter to select another default storage that has enough free disk capacity to store the virtual disk(s).

Other issues

  • Some errors are not reported in detail by XEN API. More information can be found in XenServer in /var/log directory.
  • In some pool configurations all pool members must have their management interface listed in DNS.
  • If the settings below don't agree on asynchronous routing and connection drops and/or 3 second or 80 second delays, opening sockets may occour. This is very similar to having 2 network cards configured with ipv4 to the same subnet and connected to the same switch.
  • Most network related performance and reliability issues can be resolved with Troubleshoot networking (XenServer).
  • In some situations when using thin storage repositories, an sr rescan is necessary to re-detect free space before a backup or restore.

SAP HANA

After initial configuration the first test backup out of the HANA Studio shows errors

Problem

  • Once you configured SAP HANA to work with SEP sesam and start the first test backup, it finishes with errors.

Cause

  • There might be an issue with case sensitive file paths, written in the /var/opt/sesam/var/ini/backint_saphana.utl files.

⇒ Solution

  • Open the /var/opt/sesam/var/ini/backint_saphana.utl files and make sure that the directory names are listed in the appropriate case (lower case or upper case respectively).

SAP HANA backup fails as sbc_hana.sh tries to back up SAP HANA DB on a secondary system

Problem

  • sbc_hana.sh does not recognize the secondary SAP HANA server as the secondary system and attempts to back up the SAP HANA database. Backup fails with error Connection refused.

Cause

  • The secondary SAP HANA DB cannot be backed up as it permanently synchronizing data with the primary SAP HANA DB. Only the primary system is able to access the database when the database is running in SYNC mode. Therefore accessing the secondary system and running the backups on the replication target will fail.

⇒ Solution

Low deduplication ratio for SAP HANA backups

Problem

The deduplication ratio for SAP HANA V2 backups is very low or 0%.

Cause

This issue occurs because SAP has enabled backup encryption as the default setting during new installations with HANA 2.0 SP7. When backup encryption is enabled, it significantly reduces the effectiveness of deduplication processes on the backup target.

⇒ Solution

To address this issue, you have two options:

  • Disable backup encryption in SAP HANA Configuration
In SAP HANA Studio, double-click an instance to open it and select the tab Configuration. Then in the global.ini locate the setting backup_encryption and set it to off.
Note that SAP HANA backup encryption is a security feature. Disabling backup encryption will improve the deduplication ratio, but may reduce the security of your backups.
  • Use a backup target without deduplication
If you want to maintain encryption, you can switch to a backup target (SEP sesam datastore or other storage system) that doesn't perform deduplication.

For more information, see SAP HANA Configuration.

Useful SAP HANA commands

Note
The provided SAP HANA commands are intended to assist with troubleshooting low deduplication ratios. To ensure that the commands are appropriate for your specific environment, consult the official SAP HANA documentation before executing any commands.

To check the encryption status of the SAP HANA databases, you can use the following command: 

hdbsql -i <instance> -d SYSTEMDB -u SYSTEM -p <password> "SELECT DATABASE_NAME, SCOPE, IS_ENCRYPTION_ACTIVE, CONFIGURATION_CONTROL FROM SYS_DATABASES.M_ENCRYPTION_OVERVIEW"

In the SCOPE column, where the value is "BACKUP", if IS_ENCRYPTION_ACTIVE is "TRUE", the SAP HANA database will encrypt the data before creating the backup. The CONFIGURATION_CONTROL column shows whether the control over the encryption is managed by the SYSTEMDB or the local database itself.

To disable backup encryption, you can use the following commands:

  • If CONFIGURATION_CONTROL is "SYSTEM DATABASE", connect to the SYSTEMDB and use:
hdbsql -i <instance> -d SYSTEMDB -u SYSTEM -p <password> "ALTER SYSTEM BACKUP ENCRYPTION OFF"
  • If CONFIGURATION_CONTROL is "LOCAL DATABASE", connect to the specific tenant database and use:
hdbsql -i <instance> -d <database name> -u SYSTEM -p <password> "ALTER SYSTEM BACKUP ENCRYPTION OFF"

Not enough streams for SAP HANA external backups

Problem

  • SAP HANA external backups can no longer be performed due to the following error:
2018-07-26 08:04:35 W007-SBC_COM[  6664]: Timeout reached, drive group HANA has no free streams anymore

Cause

  • There are not enough streams available to run SAP HANA external backups.

⇒ Solution

  • In the data store properties, check the available streams per drive (by default, 5) and adjust them to 20.

SAP HANA backups are not started anymore

Problem

  • After a while SAP HANA backups are not starting.

⇒ Solution

  • Connect to the SAP HANA server and stop the client:
  1. /opt/sesam/bin/sesam/sm_main stop
  2. ps –ef | grep sm_sbc*
  3. Search for old sm_sbc* services and kill these services.
  4. Start the SEP sesam Client again with the command /opt/sesam/bin/sesam/sm_main start.

Backup finished with error [447]

Problem

  • Backup failed with error.
YYYY-MM-DDTHH:MM:SS+00:00  P11252      1480e025c70 ERROR   BACKUP   SAVE DATA finished with error: [447] backup could not be completed, [110507] Backint exited with exit code 1, Traceback (most recent call last):
 File "/usr/local/sesam/lib/python2.7/site-packages/cx_Freeze/initscripts/Console.py", line 27, in <module>
 File "backint_saphana.py", line 124, in <module>
 File "sm_common.py", line 55, in __init__
IOError: [Errno 13] Permission denied: '/var/opt/sesam/var/log/lgc/sbc_backint_NOT DEFINED.log'

Cause

  • The hdbbackint does not have the permissions to write to the logging directory.

⇒ Solution

  • Execute chmod 755 (directory) to fix the issue.

SAP HANA backup failure – analyze log files for cause

Problem

  • Backup was unsuccessful and is shown with a red dot in either the SAP HANA Studio or the SEP sesam GUI.

⇒ Solution

  • Go to /var/opt/sesam/var/log/lgc and search for the hdbbackint_SID.log or hdbbackint_SID_log.log. These logs contain the logging information for the data connections to the SEP sesam Server and Remote Device Server (if used).

On the SEP sesam Server/RDS, <sesam_install>/var/log/sms will contain stpd_pid.log and stpd_pid.com.log files with the server-side logging information. Additional log information on the SEP sesam Server can also be found in <sesam_install>/var/log/lgc in sm_sbc_com*.log files.

Job status displays multiple running jobs

Problem

  • Even though only a single backup or restore is running, multiple jobs are displayed in the Job Status view of the SEP sesam GUI.

⇒ Solution

  • This is the usual behaviour. When performing a backup, HANA bundles the data to be saved into a few separate save sets and sends them to the SEP sesam device server. Therefore, multiple jobs will be displayed in the job status. The same in reverse applies to restores.

Log backups running too frequent

Problem

  • Log backups are processed every 15 minutes. The time among log backups should be extended.

⇒ Solution

  • Open the backup configuration in the SAP HANA Studio. On the right, there is a box where the interval can be adjusted. But be careful: The log area will grow bigger. If the log area is full, the database will stop performing transactions!

After recovery the hardware key changed

Problem

⇒ Solution

  • This is a known issue which can occur even if there is no change in the hardware and hostname with the same configuration: re-installation, restore, recovery, or rename will change the hardware key. Please contact SAP support. For details, see Hardware Key for the HANA Database.


SAP ASE

SAP ASE backup fails with Requested server name not found

Problem

  • SAP ASE backup fails with CT-LIBRARY error: ... Requested server name not found.

Cause

  • The server name hasn't been added to the interface file or the given server name does not exist.

Note

  • If source is set with {server_name} then you probably have mistyped the server name or server's port (default 5000).
  • If source is set without {server_name} then isql looks for the server specified by your DSQUERY environment variable.

⇒ Solution

  • You have to extend the source with the correct server name and if necessary also the port.
Source format [/{server_name}[:{port}]]/{database}

For example, if the source is master, you have to add the server name as the first part, e.g. /SOL/master or /SYBASE16:5000/master. 

isql -U<username> -P<password> -S<server_name>

where <username> specifies a login name of a user with access to database (login names are case-sensitive!), <password> specifies your SAP ASE password, and <server_name> specifies the name of the SAP ASE server to which to connect. isql looks this name up in the interfaces file.

If this test fails, use the dsedit utility that allows you to view and edit server entries in the interfaces file using a GUI in Adaptive Server Enterprise. For details, see SAP ASE Utility Guide: dsedit.

SAP ASE backup fails with Failed to load library 'libsepsybase.so' error

Problem

  • A backup fails with Failed to load library 'libsepsybase.so' or Failed to load library '%1' error.

Possible causes

  • The library libsepsybase.so is not copied to the SAP ASE lib directory.
  • A wrong binary library file is used.
  • The LD_LIBRARY_PATH is not set correctly.
  • The SAP ASE backup server is not running on the specified host.

⇒ Solution

  • Copy the library libsepsybase.so to the SAP ASE lib directory.
  • Check whether a necessary shared library is installed: file libsepsybase.so and how it is resolved: ldd libsepsybase.so.
sybase16:/opt/sap/ASE-16_0/lib # file libsepsybase.so
libsepsybase.so: symbolic link to `/opt/sesam/bin/sesam/libsepsybase.so'
sybase16:/opt/sap/ASE-16_0/lib # file /opt/sesam/bin/sesam/libsepsybase.so
/opt/sesam/bin/sesam/libsepsybase.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, not stripped
sybase16:/opt/sap/ASE-16_0/lib # ldd /opt/sesam/bin/sesam/libsepsybase.so
       linux-vdso.so.1 =>  (0x00007fffcd75c000)
       libdl.so.2 => /lib64/libdl.so.2 (0x00007f65d0fa6000)
       librt.so.1 => /lib64/librt.so.1 (0x00007f65d0d9c000)
       libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f65d0b7f000)
       libc.so.6 => /lib64/libc.so.6 (0x00007f65d0803000)
       /lib64/ld-linux-x86-64.so.2 (0x00007f65d1cf3000)
  • Check if the LD_LIBRARY_PATH variable is set correctly in the file SYBASE.env (The SAP shared libraries are located in the installed component’s lib directory).
sybase16:/opt/sap # grep LD_LIBRARY_PATH SYBASE.env
[...]
LD_LIBRARY_PATH=/opt/sap/ASE-16_0/lib:$LD_LIBRARY_PATH
  • Check if the SAP ASE backup server is running on the specified host -> check for the process named backupserver.
sybase16:/opt/sap # ps -ef | grep backupserver
root     15534 15533  0 Mar23 ?        00:00:00 /opt/sap/ASE-16_0/bin/backupserver -e/opt/sap/ASE-16_0/install/qsstor.log -N25 -C20 -I/opt/sap/interfaces -M/opt/sap/ASE-16_0/bin/sybmultbuf -Sqsstor

SAP ASE backup fails with ERROR: No Total: ... line found in LIS file

Problem

  • SAP ASE backup fails with the following error: No Total: ... line found in LIS file.

Cause

  • sm_backup tried to fetch the LIS file before the backup server has finished.

⇒ Solution

  • Use the SEP sesam Client ≥ Beefalo V2 SP2 and set the following backup option in the SAP ASE backup task properties -> Options tab -> Backup options field:
-a action=sleep:20
Note
If the backup option -a action=sleep:xx does not work, use the FTP interface instead of HTTP.


SAP ERP

Using XUSER MaxDB tool for authentication during backup fails with the error XUser not found!

Problem

  • The SAP MaxDB XUSER command-line tool can be used to store user log-on data thus simplifying logon to databases. When using XUSER and the script runs as command event, the backup script reports the following error: XUser not found!

Cause

  • The SAP MaxDB XUSER tool creates a file .XUSER.62 in the home drive of the corresponding user. If the script runs as command event, no environment is set and thus also not the HOME variable. If the HOME variable is not set, the DB cannot find the .XUSER.62 file belonging to the user and cannot map and verify the key to the user, therefore the command event fails.

⇒ Solution

  • Export the HOME variable for the root user in the backup script (command event script) as follows:
#!/bin/bash
HOME=/root
export HOME
/.../dbmcli .... -U <key> ....
.....


NetApp Volume Backup

FilerView http might be disabled

Problem

  • FilerView might be disabled.

⇒ Solution

  • Check if FilerView is enabled. Logon to your NetApp system via ssh and use the following command to check whether your FilerView http is enabled:
options httpd.admin.enable

In clustered environments or Ontap 8 Versions using Vserver: 

vserver services web modify -vserver <NODE_NAME> -name ontapi -enabled true

Restore shows permission problems

Problem

  • A restore fails with the following error message in the protocol:
2012-08-20 12:32:27: sbc-2044: Warning: Cannot create item 
[/var/opt/sesam/var/work/mnt/netapp/VOLUME_NAME/restore/]: Permission denied

⇒ Solution

  • Check if the system acting as a data mover has a write access to the NetApp volume the data should be restored to.

Incorrect volume path specification

Problem

  • Usually NFS shares on NetApp systems are exported beneath the /vol/ directory. For example the nfs share test is mountable via: netapp:/vol/test/

This is not the case for all NetApp systems. Some systems export the volume directly via: netapp:/test/

⇒ Solution

  • By default, SEP sesam mounts the volume via /vol/ specification. You can adjust the volume_path parameter in the advanced backup options by specifying the following:
-a volume_path=/

Backup fails with Permission denied - user mapping for NTFS style volumes is missing

Problem

  • By default, SEP sesam is accessing the volumes for backup via NFS. If a desired volume for backup has security style set to NTFS, an appropriate user mapping has to be configured on the NetApp system. If not, backup may fail with Permission denied error message.

⇒ Solution

  • Map the permission for the administrator user to root as shown in the screenshot below.



MySQL Troubleshooting/en

BSR Pro for Windows

BSR Pro error codes

When performing a BSR Pro backup or restore, the procedure might fail due to different reasons. The following list may be of help to understand the BSR Pro error codes.

  • e0040002: There is a BSR Pro backup or restore currently running; simultaneous operation of more than one BSR Pro backup or restore process is not supported.

BSR Pro installation stops with a message: A new version was found. An update is not necessary.

Problem

  • BSR Pro installation fails with a message, such as: A new version was found. An update is not necessary. This happens when the installation procedure has found another installed version of the BSR Pro (or the original program DiskImage from O&O).

Possible cause

  • In the past, the BSR Pro or O&O DiskImage was already installed manually.

⇒ Solution

  • If you want to keep the currently installed version, you have to perform the SEP sesam installation without the BSR Pro. Otherwise the currently installed BSR Pro version has to be uninstalled and then the SEP sesam installation together with BSR Pro has to be performed again.

BSR Pro fails to update during silent SEP sesam update

Problem

  • BSR Pro is not updated when you update a SEP sesam Server, RDS or Client component including BSR in silent mode, e.g. via the powershell command line:
sesam-cli-5.1.0.3-windows.x64.exe -Wait -Verb runAs -ArgumentList '/s',/v"/quiet /lvoicewarmup \"C:\tmp\sm_msi_update.lgc\""

Cause

  • BSR Pro is installed with own MSI installer and cannot be included automatically in a chained installation. During a manual update using the installer EXE, or during the remote update installation via the SEP sesam GUI (sm_update_client) BSR Pro will be updated normally. During the silent installation the update of BSR Pro is not performed.

⇒ Solution

To update the BSR Pro you can perform one of the following:

  • Use the regular GUI-based update which also includes the updated version of BSR Pro.
  • Use the SEP sesam GUI (sm_update_client) update which also includes BSR update.

BSR Pro backup fails due to snapshot backup issue

Problem

  • BSR Pro backup fails with BSR Pro unknown return code (0X1).

Cause 1

  • On some Windows OS (e.g., Windows Server 2012) you may receive a "volsnap" error in the Windows Event log: Event ID 25: The shadow copies of volume C: were deleted because the shadow copy storage could not grow in time. Consider reducing the IO load on the system or choose a shadow copy storage volume that is not being shadow copied.

This error indicates that the source drive is experiencing a high IO load, which caused VSS to fail and deleted the shadow copy snapshot, as a result causing a BSR Pro backup failure.

⇒ Solution 1

  • You have to reduce the IO load on the drive being backed up or configure VSS to use a different drive with more available space (and less load) and ensure that there is sufficient storage space to create and maintain shadow copies. You may want to allocate a separate volume on a separate disk for storing shadow copies, as recommended by Microsoft, to improve performance and eliminate the possibility to delete the shadow copies due to a high I/O load.

For details, see Volume Shadow Copy Volsnap Event 25. See also the following Microsoft article: Event ID 25 — Diff Area Integrity

Cause 2

  • If the BSR backup fails with error message 0X1 without a clear cause, it is often due to an inconsistency in the client's operating system.

⇒ Solution 2

  • To verify this, check the Event Viewer on the client being backed up. The time of the error usually aligns with the errors logged in the BSRPro.log file. This log is located on the client in <SEPsesam>\ProgramData\SEPsesam\SEP sesam BSR Pro\.
  • The easiest way to resolve the inconsistencies is to restart the client.

No savesets nor clients exist in SEP sesam after clicking Execute BSR Pro Quick-Start option

Problem

  • The list of available save sets at the SEP sesam Server remains empty.

Possible cause

This error typically occurs when the connection to the SEP sesam Server cannot be established due to any of the following:

  1. Problem with name resolution: SEP sesam Server hostname is not known to client, name to IP or IP to name (reverse) resolution is not correct.
  2. The client's name (typically a new host is set up in case of disaster recovery with new IP) cannot be resolved at the SEP sesam Server.
  3. Wrong server name (a fully qualified domain name (FQDN) may be required).
  4. When booting from PE, it may take a little longer until the savesets appear (30-45 seconds).

⇒ Solution

  1. Check your name resolution on client and SEP sesam Server. Both names must be resolved, the client IP must be resolved to the client name, and SEP sesam Server must be reachable (check the route to it – FQDN may be required to reach it). For reference, see How to check DNS configuration. If the resolution in your network does not work as expected, there might be issues with IPv6 and IPv4.
  2. Add the client's name to the name resolver or add it to the /etc/hosts on SEP sesam Server, or use the GLBV gv_stpd_auth NONE to disable the IP name to IP address check on SEP sesam Server; in the latter case, open a shell on the SEP sesam Server and execute the command below. This command enables you to contact the SEP sesam Server even if the reverse name resolution does not work on the new client.
    3.  sm_glbv w gv_stpd_auth NONE
  3. Specify the SEP sesam Server with the FQDN. Open the BSR Pro Settings -> tab Security and in the Authentication table check the settings. Under the Network path/server, a valid sesam:{server} name or interface must be specified.
    4. Examples: 
     ''Network path/server''                      ''Port''
 sesam:http://sep_server.mydomain.com   11000
 sesam:sep_server2                        11001
    Note
    Only the first entry is used to fill the list of available savesets!

OODIAG crashes during the creation of the BootISO

Problem

  • If you try to create a boot ISO with the BSR Pro on a system where the volumes are dynamic volumes, then the OODIAG service cored under certain circumstances.

Possible cause

  • Problem is apparently the system volume. Possibly it is related to the dynamic volume.

⇒ Solution

  • Install a newer SEP sesam version with BSR Pro.
  • If possible, do not use dynamic volumes.

VSS snapshot for the system volume does not work

Problem

  • Typically, VSS is used by default to log changes in data so that unchanged data is included in the backup. But on some systems the VSS snapshot for the system volume does not work.

⇒ Solution

  • Start SEP sesam BSR Pro with the option -a snap=snapshot. This option creates a special snapshot that does not involve the Microsoft VSS Writer: data changes are logged using the installed filter driver to include the unchanged data in the backup.
Usage: -a snap=vss|snapshot
where <vss> is default

It is not possible to get the error-log for a failed restore in a PE

Problem

  • Disk full while accessing X: because dump is written to it.

Possible cause

  • The LOG is getting too big because it is constantly being written on.

⇒ Solution

  • Open the registry in the command shell by using regedit.
  • Add the following reg-key:
[HKEY_LOCAL_MACHINE\SOFTWARE\O&O\O&O DiskImage\<version>]
"DumpFilePath"="E:\\DUMPS\\"
  • After the change the LOG is written to the path E:\DUMPS.

SEP sesam uninstallation of the BSR Pro component fails

Problem

  • Uninstallation of the BSR Pro component fails and leaves invalid registry entries behind.

⇒ Solution

  • After a failed uninstallation, the uninstaller information is probably corrupted and you have to get rid of any potentially harmful registry leftovers to ensure normal operation of SEP sesam. There are two possible ways to deal with failed uninstallation leftovers.
  1. The best way to uninstall these components is to use the installer; you should first reinstall BSR Pro in order to repair it, and only then uninstall it again.
    2. Tip
    After a successful and complete uninstall of the BSR Pro component, the keys listed below (see registry entries) should no longer exist. If any of the listed keys is still present, it can be deleted manually as described in the next procedure.
  2. If the procedure above cannot be successfully applied, you have to manually remove the BSR Pro installation.
    3. Note
    The following steps describe how to modify the registry. If you modify the registry incorrectly, serious problems might occur. If you are not sure about what you are doing, we recommend that you contact SEP support at support@sep.de for assistance.
    1. In the Start menu/Search box, type regedit and click Enter. The Windows Registry Editor window opens.
    2. Delete the registry entries:
      3.  HKEY_LOCAL_MACHINE\SOFTWARE\O&O\O&O DiskImage
 HKEY_LOCAL_MACHINE\SOFTWARE\O&O\O&O LiveUpdate\SEP sesam BSR Pro

BSR Pro backup fails with a connection error

Problem

  • BSR Pro backup may fail with one of the following errors due to blocked network connection to SEP sesam Server:
script processing failed:

or 

(Status: ERROR 425 Can't open data connection. WINSOCK: Connection timed out. (0x274c,10060)

or 

Port negotiation failed. (10038)

Possible cause

  • Windows Firewall is preventing or blocking your connections.

⇒ Solution

  • Check the Windows Firewall rules on the SEP sesam Client with BSR Pro. If the firewall rule applied during SEP sesam installation is no longer valid, create an exception for oodiag.exe to allow connection through the firewall. For details on how to configure this firewall rule, check the Prerequisites in the SEP sesam BSR Pro – Backup Configuration.


NDMP

NDMP connection error during backup

Problem

  • The NDMP server fails to connect to SEP sesam (sbc_ndmp) during an NDMP backup. Errors like the following are displayed and the backup fails.
NDMP error:  ERR NDMP4_DATA_CONNECT  NDMP4_CONNECT_ERR
NDMP error:  ERR NDMP4_DATA_ABORT  NDMP4_ILLEGAL_STATE_ERR

or 

NDMP error: NDMP_DATA_CONNECT

Possible causes

  • In most cases, this is a firewall problem.
  • Other incorrect network settings prevent the connection to sbc_ndmp.
  • In the case of NetApp, multiple interfaces may prevent the connection to sbc_ndmp.

⇒ Solution

  • Exclude sbc_ndmp.exe from the firewall scanner and run the backup again.
  • Check the configuration of your network. Note that your network must be configured correctly to allow the connection to sbc_ndmp.
  • In the case of NetApp, if multiple interfaces are suspected, set the preferred interface option on your NAS device to specify the interface which should be used for NDMP:
options ndmpd
options ndmpd.preferred_interface <interface_name>

For more details, see NetApp-specific NDMP configuration.

NDMP restore to new restore target with non-existent subdirectory fails

Problem

  • An NDMP restore to a new restore target where the intended restore target is a different, non-existent subdirectory fails. (In previous SEP sesam versions, the NDMP alternate restore to a non-existent subdirectory fails, but it is shown as successful. Although this is now fixed, you may want to verify that the previously restored NDMP data exists.)

Note that this is only the case if the new restore target is a non-existent subdirectory. For example, if you want to restore to a /<volume_name>/<sub_dir1>/<sub_dir2> and <sub_dir1> does not exist, the restore will fail. You can still restore NDMP to the alternative target subdirectories if they already exist.

Cause

  • If a subdirectory is specified as the new (non-existing) restore target, for example, /<volume_name>/<sub_dir1>/<sub_dir2>, the restore fails as non-existing subdirectories are not supported when restoring to another target. In this case, only a single level directory structure can be used as the restore target: /<volume_name>/<target_dir>; so in the above example, the supported restore target path is /<volume_name>/<sub_dir1>.

Workaround

  • When performing an NDMP restore to a new restore target, make sure that you restore your data to the top-level directory (/<volume_name>/<target_dir>). Do not specify any non-existing subdirectories in the new target path, otherwise the restore will fail.

Selective NDMP restore does not restore empty directories and the restore fails

Problem

  • In SEP sesam versions ≤ 4.4.3 Beefalo V2, when performing a selective NDMP restore, empty directories are not restored (created) on the target if:
    • empty directories are present in the restore list along with normal directories and files (this is a known issue, but too minor to warrant a fix)
    • at least one file is selected in another directory, for example, if the backup source is /vol/vol1 and the restore selection is /vol/vol1/dir_1/file_1 and /vol/vol1/empty_dir

Consequently, the restore fails.

Cause

  • If only directories are selected for selective restore, the empty directories are added to the SEL file by sm_restore and the restore completes without error. However, if at least one file is selected together with an empty directory, the restore fails.

Workaround

  • When performing a selective restore, try not to restore empty directories together with individual files, otherwise the restore will fail.

NDMP on NetApp

NDMP browsing of non-UTF-8 fails and restore to non-UTF-8 volume may not be possible

Problem

  • When performing a restore, it is not possible to browse the contents of non-UTF-8 NetApp volumes. You may receive an error message similar to the following:
NDMP error:  ERR NDMP4_SNAP_DIR_LIST  ?0x20500106?

This can happen because:

  • Non-UTF-8 volumes are not searchable if objects contain umlauts or special characters.
  • Consequently, a restore to a non-UTF-8 volume is not possible if objects with special characters are included in the NDMP backup.

The following ONTAP CLI command displays the encoding language of all volumes: 

volume show -vserver * -fields language

⇒ Solution

  • NDMP backups of NetApp volumes without UTF-8 encoding language are basically possible, but the path in the backup task can only be specified manually.
  • If a backup of a non-UTF-8 volume contains objects with umlauts or special characters, the restore can only be performed to a UTF-8 volume.

NDMP DAR restore of single files is too slow

Problem

  • When using a Direct Access Recovery (DAR) which enables NDMP selective restore, restore is too slow, even though DAR should greatly reduce the time it takes to restore individual files.

Cause

  • The general prediction is that DAR greatly increases restore speed, but this depends on the amount and size of data backed up. As stated by NetApp, the restore can take quite some time depending on the number of files in the directory, the position of the directory and file on the tape, and the number of tapes to be read. Refer to the NetApp documentation for details.

⇒ Solution

  • Split the backup into smaller parts to reduce the amount of information that DAR has to manage during the restore.

Restore fails with: "Storing of nlist entries failed."

Problem

  • Restore finishes with the error message: "Storing of nlist entries failed."

⇒ Workaround

  • Instead of commonly used version 4 of the NDMP protocol, start the NDMP daemon in version 3 by entering the following commands:
ndmpd off
ndmpd version 3
ndmpd on
  • Then retry the NDMP restore.


Hyper-V Troubleshooting/en

KVM/QEMU

Merging and deleting leftover snapshots

Problem

  • The backup fails and the snapshot is left behind.

⇒ Solution

Use the virsh utility, as shown in the example:

  1. List the available snapshots for the domain:
    2.  user@hypervisor:~$ virsh snapshot-list <domain_name>
 Name                 Creation Time             State
 ------------------------------------------------------------
 Sesam_SF20173828282@XXXX      2017-07-06 08:15:11 +0200 disk-snapshot

    In this example, one leftover snapshot for this VM exists.

  2. List the virtual disks for the domain:
    3.  user@hypervisor:~$ virsh domblklist <domain_name>
 Target     Source
 ------------------------------------------------
 sda        /path/to//Sesam_SF20173828282@XXXX.snapshot
  3. For each device that refers to SEP sesam snapshot, start a block commit to merge the snapshot:
    4.  user@hypervisor:~$ virsh blockcommit <domain_name> sda --active --verbose --pivot
 Block Commit: [100 %]
 Successfully pivoted
  4. Confirm that the device is now switched to the original disk device:
    5.  user@hypervisor:~$ virsh domblklist <domain_name>
 Target     Source
 ------------------------------------------------
 sda        /my/original/base.img
  5. Delete the snapshot metadata information:
    6.  user@hypervisor:~$ virsh snapshot-delete <domain_name> --metadata --snapshotname Sesam_SF20173828282@XXXX
 Domain snapshot sesam_snapshot deleted
  6. Delete the snapshot file:
    7.  user@hypervisor:~$ rm /path/to//Sesam_SF20173828282@XXXX.snapshot


Si3 Deduplication

Failed backups to Si3 data store

Problem

If the system running the SEP sesam Server is suspended and an attached hard disk is hibernated, backups to an Si3 data store located on that disk fail. However, backups to a standard data store may continue to function as expected.

In the log files the following entry may be found: 

Drive 27:
  14:51:08 Automatic backup 0-01gb-test completed
  14:51:08 553 STOR Failed. error in write_http: Failed to send chunks. Error: {“code”: “UNEXPECTED_SERVER_ERROR”, “message”: “java.io.IOException: The device is not ready”} error in finish_http: Unexpected error on session abort, sid: 103 . Error: {“code”: “SESSION_NOT_FOUND”, “message”: “103”}

Cause

The Si3 data store requires continuous access to the underlying storage system. When the attached hard disk is hibernated or the system is suspended, the storage becomes unavailable.

⇒ Solution

Verify that the disk hosting the Si3 datastore is fully operational and accessible. Ensure that the attached hard disk is not set to hibernate and avoid suspending the system during backup operations.

Unable to establish connection to S3 data store

Problem

Si3 data store may be unable to establish secure connection to S3 storage with the following error: 

Error: Could not access data store. Server Status: 2023-03-30 10:17:10: ERROR Not started due to error: S3 is not connected Server Status: 2023-03-30 10:17:10: ERROR Not started due to error: S3 is not connected

Cause

In case Si3 data store connects to a storage provider that uses a self-signed certificate, this certificate is not recognized as trustworthy by default because it is not issued by a trusted certificate authority. This can result in connection being denied and log files in /var/opt/sesam/var/log/sms may contain a log message similar to this: 

[...default-dispatcher-6] [1;31mERROR[0;39m [36mS3[0;39m - Unexpected error: {}, cause: {}
software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem

⇒ Solution

To solve this problem use the keytool utility to import the public.crt certificate to the server certificate store. This will allow the Si3 server to recognize and trust the S3 storage provider's certificate, and establish a secure connection.

  1. Obtain the public certificate. Note that you can export it from the browser.
  2. Locate the cacerts file on your server. This is the location of your JVM certificate keystore.
  3. Import the public.crt certificate into the JVM's certificate keystore with the following command:
on Linux:
keytool -import -trustcacerts -keystore /var/lib/ca-certificates/java-cacerts -storepass changeit  -noprompt -alias <storage backend endpoint URL> -file /<path_to_certificate>/public.crt
on Windows:
C:\Program Files\ojdkbuild\java-11-openjdk-11.0.15-1\bin>keytool -import -trustcacerts -keystore "C:\Program Files\ojdkbuild\java-11-openjdk-11.0.15-1\lib\security\cacerts" -storepass changeit -noprompt -alias <storage backend endpoint URL> -file <path_to_certificate>\public.crt

Issues with S3 or S3-compatible storage

Problem

  • Si3 data store using S3 or S3-compatible storage can experience various issues, depending on cloud storage provider. These issues can affect backups, migrations, and replications. In addition, sanity state check of Si3 could report errors that have similar root cause.

Cause

  • Some cloud storage providers (for example, Wasabi) have request rate restrictions (how many HTTP(S) requests are allowed per second). Also on local storage with S3 option enabled, when multiple RDSs access the same local S3 storage, this can generate a lot of IOPS (I/Os per second).

⇒ Solution

  • You can adjust the settings on the affected Si3 data store:
  1. In the Main selection -> Components, click Data Stores to display the data store contents frame.
  2. Right-click the selected Si3 data store and then click Properties.
  3. Double-click a drive to open Drive Properties dialog, and then in Options field enter as follows:
dedup.s3.timeoutInSeconds=1200,dedup.s3.page.workers=2,dedup.maxAsyncRequests=50
This will increase the timeout period, active page workers and request rate.

Si3 remains in "shutting down" state

Problem

  • Manually stopping Garbage Collection (GC) fails and consequently Si3 remains in the "shutting down" state.

⇒ Solution

  • Restart the Si3 daemon by using sm_main restart sds. For more details on stopping and starting the SEP sesam services, see How to Start and Stop SEP sesam.

Si3 deduplication may not work with NFSv4

Problem

  • Si3 deduplication may not work with Network File System version 4 (NFSv4).

Cause

  • SEP sesam operations, such as backup, restore and migration, may fail due to Java problems with NFSv4.

⇒ Solution

  • To avoid this problem, connect your backup devices via NFSv3.

Repairing corrupted Si3 NG data store

You can repair the Si3 store when pages or objects get corrupted.

  1. First determine the scope of corruption:
    • To get the list of corrupted objects use: 
      sm_dedup_interface -d <datastore> corruptedobjects
    • To get the list of corrupted pages use: 
      sm_dedup_interface -d <datastore> corruptedpages
  2. Use the following command to replace the page in /pages directory with an older version from /pages-trash directory: 
    sm_dedup_interface -d <datastore> repair pages
    The pages in trash contain all chunks deleted on previous GC. The oldest version of a page takes priority.
  3. Use the following command to search for and recover the missing chunks in /pages-trash directory: 
    sm_dedup_interface -d <datastore> repair start
    During the repair process a new page is created, which contains all chunks from the current page (page affected by 'missing chunks' issue) and all chunks found in the trash.

Cleanup of unrecoverable Si3 store

Warning
You should use the commands described in this section only in case the corrupted store cannot be recovered.

When corruptions in the Si3 store persist, the initial page version has already been purged from trash or there were fatal errors during backup or restore. In this case broken pages or missing chunks cannot be recovered.

Cleanup can be performed by deleting unrecoverable objects manually or by using the automatic cleanup function.

Deleting objects

When there are only a few unrecoverable objects, delete each object with the following commands: 

sm_dedup_interface -d <datastore> delete corruted_object_id_1

...

sm_dedup_interface -d <datastore> delete corruted_object_id_Nth

In case of many corruptions you can delete all corrupted objects using the following command: 

sm_dedup_interface -d <datastore> fsck purge
Garbage collection

When you have deleted all unrecoverable objects, run garbage collection (gc): 

sm_dedup_interface -d <datastore> gc start
Automatic cleanup function

To start an automatic cleanup function, use the following command: 

sm_dedup_interface ... fsck purge auto

The automatic cleanup function runs the following sequence of commands: PCCK start -> OCCK start -> Delete all corrupted objects -> GC start.

Logging

The logging function uses a relatively powerful logback library. For more information, see Logback Project. Note that this information is intended for advanced users only.

Logging info
  • gv_rw_ini:sm_sds.xml (/var/opt/sesam/var/ini/sm_sds.xml)
  • /var/opt/sesam/var/log/sms contains two log files:
    • sm_dedup_server_info-<drive>.log: Log level INFO and higher.
    • sm_dedup_server-<drive>.log: Log level DEBUG and higher. This file can become quite large.
    • sm_dedup_gc-<drive>.log: garbage collection log.
    • sm_dedup_fsck-<drive>.log: file system check log.
  • Auto rotation if the log file size reaches 100 MB.

Files and directories

Objects

For every SEP sesam saveset, three objects (files) are stored in the Si3 store:

  • <ssid>.data
  • <ssid>.info
  • <ssid>.info2

The .data and .info files are identical to those of a normal data store. The .info2 file is required for the data to be appended to a Si3 object. All database information that is not available before a backup is completed is written to this file.

Directories


Tape and Tape Devices Troubleshooting/en VSS Troubleshooting/en

HPE StoreOnce Catalyst

HPE StoreOnce backups fail

If your backups fail, one of the reasons could be related to HPE StoreOnce sizing parameters. In this case, check the following:

  • If you have defined a Physical or Logical Storage Quota for your Catalyst store, check if the quota limits have been reached. If so, increase the quota to a sufficient size. For details, see HPE StoreOnce Configuration.
  • If you created an HPE Catalyst data store in SEP sesam and later changed the HPE Catalyst store size parameters, e.g., by changing or removing the storage quotas, check their values in SEP sesam GUI: Main selection -> Components -> Data Stores, double-click your StoreOnce store, and then click the HPE Catalyst Store State tab. If the data store status is set to "failed", you may need to adjust the StoreOnce data store (Size) Capacity and High watermark values to allow for correct calculation and make the data store functional again. For details, see Size and Disk space usage.

HPE StoreOnce backup failed with "553 STOR Failed. Error: Ctl_OpenObject failed, error: OSCLT_ERR_MAXIMUM_SESSIONS. (0)"

Problem

HPE StoreOnce backup failed with the following error message: 

553 STOR Failed. Error: Ctl_OpenObject failed, error: OSCLT_ERR_MAXIMUM_SESSIONS.

This issue can occur when an HPE StoreOnce Catalyst data session fails to open because the maximum number of sessions on the HPE StoreOnce Catalyst server is reached, thus no other session can be started until resources are available. You can check the number of free data sessions in SEP sesam by opening your HPE StoreOnce data store properties and clicking the tab HPE Catalyst Store State, then checking the value of Free data sessions.

⇒ Solution

There are two possible solutions:

  • The HPE Catalyst server supports a limited number of concurrent data sessions. Therefore, you need to ensure that the maximum number of data sessions for your HPE StoreOnce is not exceeded, or increase the maximum number of data sessions (including restores) that can be run simultaneously.
  • Open SEP sesam and from the Main selection -> Components -> Data Stores double-click your HPE StoreOnce data store to open the properties. Then, under Drives click the relevant drive to view its properties. In the Max. channels drop-down list, decrease the number of available channels for concurrently running backup/migration streams. By default, the number of available channels is set according to your SEP sesam Server license (e.g., the standard license supports 10 concurrent streams, enabling 10 backup processes to run simultaneously).

HPE StoreOnce backup failed with error code "OSCLT_ERR_SERVER_OFFLINE"

Problem

A backup to a HPE StoreOnce datastore failed with the following error message: 

553 STOR Failed. Command error with HPE StoreOnce server [<hostname>]: OSCLT_ERR_SERVER_OFFLINE. (0)

⇒ Solution

  • Ensure that the SEP sesam Client can successfully connect to the HPE StoreOnce storage system over the network using the designated data transfer method.
  • Check the DNS configuration on the SEP sesam Client to ensure it can resolve the storage system's hostname.
  • Check network connectivity, correct routing, and that any required protocols or ports for communication are configured and accessible. Additionally, check the firewall settings along the network path to ensure no traffic is being blocked and that the correct routes are in place.


Proxmox VE

Proxmox VE backup does not work with the client name as plain IP address

Problem

  • If the node added to the SEP sesam environment is not set up correctly and an IP address is used as the client name instead of the client's hostname matching the hostname returned by the Proxmox server, the backup fails.

⇒ Solution

Proxmox VE backup failed with a connection error

Problem

  • Proxmox VE backup failed with Error connecting to proxmox System: "('Connection aborted.', gaierror(-2, 'Name or service not known'))"]. This error occurs after a VM has been migrated to another cluster node. Consequently, the Proxmox VE backup fails.

⇒ Solution

  • Ensure that every Proxmox VE cluster node can correctly resolve other cluster nodes using DNS or the Hosts file.

File backup of Proxmox hypervisor fails

Problem

  • Backup of /etc/pve/ fails.

⇒ Solution

  • Exclude the /etc/pve/ directory from backup. /etc/pve/ is just a file system representation of the cluster database, which is located under the path /var/lib/pve-cluster/config.db. For more information, see the Proxmox wiki article. For more information, see also Proxmox Cluster File System (pmxcfs).


See also

Error Messages Guide

Retrieved from "https://wiki.sep.de/wiki/index.php?title=Troubleshooting_Guide&oldid=219121"