Disaster Recovery Troubleshooting

From SEPsesam
Jump to: navigation, search


Copyright © SEP AG 1999-2020. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Docs latest icon.png Welcome to the latest SEP sesam documentation version 4.4.3/4.4.3 Beefalo V2. For previous documentation version(s), check documentation archive.


Disaster recovery on Linux

Problem with a ReaR backup execution

Problem

  • SEP sesam Client package was successfully installed and path backups are working. If you have problems executing a ReaR backup check the backup log if there are some missing dependencies.
  • Example:

    There are binaries or libraries in the ReaR recovery system that need additional libraries
    /opt/sesam/bin/sesam/libvirtmod.so requires additional libraries (fatal error)
    	libvirt.so.0 => not found
    

Cause

  • In this case the package libvirt-client is missing.

Solution

  • Depending on your distribution, install libvirt-client as follows:
    • On RHEL/CentOS 7 use the command: yum install libvirt-client
    • On SLES 12/15 use: zypper install libvirt-client

The workflow mkrescue is not supported in the ReaR system

Problem

  • The workflow mkrescue is not supported in the ReaR rescue/recovery system.

Solution

  • Delete the file /etc/rear-release.

ReaR image hangs during bootup

Problem

  • The system hangs during bootup like shown in the following image:
  • Rear-hang.jpg

Solution

  • Boot the system with the ACPI=OFF option (this option can be specified on the command line in the boot menu prompt, after the options BACKUP=SESAM OUTPUT=ISO).

The recovered system does not boot

Problem 1

  • The system does not boot because /root/dev/console cannot be found.

Possible causes

  • Certain distributions rely on the existence of the directory /dev/ while booting
  • Certain static devices must exist before the udev daemon creates them.

Solution

  • Include the /dev/ file system in your backup.
  • If the restore cannot restore /dev/:
  1. Boot from the SEP sesam LIVE CD
  2. Mount the ROOT partition of the restored system
  3. Manually create the /dev/ directory
  4. Manually create the /dev/console entry with:
mknod /path/to/target/mount//dev/console c 0 0

Problem 2

  • The system does not boot because of missing libblkid.so.1.

Possible cause

  • This is most likely caused by SELinux which is activated by default.

Solution

  • Especially on RHEL6 or CentOS6 systems, follow these steps after rebooting from the ReaR recovery:
    1. Press a key when prompted by the boot loader (GRUB):
    2. Rhelcentos grub1.jpg

    3. Select the appropriate boot loader entry:
    4. Rhelcentos grub2.jpg

    5. Press e to modify the commands for the selected entry:
    6. Rhelcentos grub3.jpg

    7. Add selinux=0 to the commands:
    8. Rhelcentos grub4.jpg

    9. Press Enter to confirm the changes and b to boot up the machine with SELinux disabled.
    10. When having access to the system, change the option SELinux of /etc/selinux/config to the following:
    11. SELINUX=permissive
      

    Afterwards, reboot the system and feel free to set the SELinux value back to enforcing if needed.

No bootable operating system can be found

Problem

  • The system is not able to find a bootable OS instance after the restore.

Possible causes

  • There may have been problems during the installation of the GRUB boot loader.

Solution

  • The restore protocol includes a statement whether or not the installation of the boot loader was successful:
2009-12-14 14:48:27: sbc-3500: Info:     Reinstall boot manager
[/sesam/bin/sesam//sbc_grub_auto /mnt/disk/ AUTO]

  • It is also possible to boot the system again from the live-CD, mount the target partitions and use grub-install to install the boot loader correctly.

The device does not have a corresponding BIOS drive

Problem

  • During the restore, the following error occurs:
/dev/sda1 does not have any corresponding BIOS drive

Possible causes

  • Check the file /boot/grub/device.map on the target system. If there are entries referring to the disk through /dev/by-disk/... as shown in the example below, the entry is most likely the reference to the hard disk partition of the broken system. GRUB will not find the proper device:
hd(0) /dev/disk/by-id/ata-SAMSUNG_SP2504C_S09QJ1GLA14263-part1

Solution

  • Reboot from the live-CD
  • Mount the root and boot partitions to /mnt/disk (and /mnt/disk/boot, if necessary)
  • Restart grub-install with the following options:
grub-install --root-directory=/mnt/disk --recheck hd0

Output:

grub-probe: error: Cannot open `/boot/grub/device.map'
/usr/sbin/grub-install: line 374: [: =: unary operator expected
Installation finished. No error reported.
This is the contents of the device map /mnt/disk/boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.

(hd0)   /dev/hda
(hd1)   /dev/hdb

You can ignore the error line 374: [: =: unary operator expected.
More important is the result Installation finished. No error reported.

No corresponding BIOS drive for /dev/cciss/c0d0p2

Problem

  • You receive the message: /dev/cciss/c0d0p2 does not have any corresponding BIOS drive in restore log.

Solution

fsck.ext3: File system has unsupported features

Problem

  • During a restore of a system with kernel version 2.4 the system may not boot because the Live-CD creates a file system with features which are not supported by kernel 2.4.

Possible causes

  • Most likely the file system options resize_inode,dir_index,large_file,ext_attr are causing the problem and making the system unbootable.

Solution

  • Reboot from the Live-CD image, which includes the tool debugfs.
  • Show the file system features with debugfs:
root@recover#: debugfs -w /dev/sda2
debugfs 1.41.1 (01-Sep-2008)
debugfs:  features
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
quit

Replace /dev/sda2 with the corresponding partition names on your system.

  • To remove file system features:
root@recover#: debugfs -w /dev/sda2
debugfs:  features -resize_inode -ext_attr -dir_index -large_file -needs_recovery -sparse_super
Filesystem features: has_journal filetype
quit

After removing the options, the system should boot correctly.

Incorrect inode size (256)

Problem

  • After a successful restore the boot process stops with incorrect inode size (256).

Possible causes

  • Older kernel versions (2.4) may use a different inode size than the one the file system's created through the Live-CD (which includes kernel 2.6). For example, this happens during the restore of SLES8 based systems which use an inode size of 128k.

Solution

  • This can only be solved by formatting the devices manually from the Live-CD, using the proper mkfs options:
mkfs.ext3 -I 128 /dev/sda1

After this step, remount the partition to /mnt/disk and repeat the restore operations. Changing the inode size is only possible by reformatting the devices.

Missing root file system

Problem

  • The restored system can't find a root file system and fails during resume.

Possible causes

  • The /etc/fstab file was configured with the root file system as UUID.

Solution

  • Specify the root file system device name in conventional device names if you are using a different physical disk. After booting, use YAST to reconfigure your boot loader or edit your /boot/grub/menu.lst manually:
root=/dev/sda2

Missing network cards

Problem

  • The restored system does not find any network cards.

Possible causes

  • If the restore was done to dissimilar hardware, SLES-based distributions may not configure the network devices correctly. SLES-based systems save their network configuration by using the system's MAC address. Most likely the system will not use eht0 as a device name, but eth1, as it has another MAC address.

Solution

  • Use YaST and reconfigure your network interfaces.

Client does not start on the RHEL6/Debian9 recovery image

Problem

  • The SEP sesam Client does not start automatically on RHEL6 and Debian9-based recovery images.

Cause

  • The file /etc/init.d/functions is missing within the recovery image.

Solution

  • The client can be started manually via:
  • /opt/sesam/bin/sesam/sm_main start
    

RHEL7-related issues

RHEL7 backup fails with an error

Error 1

  • The RHEL backup fails with the following error:
  • ERROR: The LSB package is not installed.
    

Solution

  • Install the lsb package as follows:
  • yum install redhat-lsb-core mkisofs syslinux
    

Error 2

Solution

  • To solve this problem, proceed as follows:
    1. Edit
    2. /var/opt/sesam/var/lib/rear/usr/share/rear/conf/default.conf
      

      and from the line:

      # required programs. Same as above, but if they are missing, we abort.
      REQUIRED_PROGS=(
      "$SCRIPT_FILE"
      

      remove the line:

      mingetty
      
    3. Run the backup again.

ReaR error occurred during grub2-mkimage of bootx64.efi

Information sign.png Note
In order to be able to create an UEFI/EFI bootable ISO image, the additional tool ebiso has to be installed on the client system as described in the section Installing ebiso for creating UEFI aware ISO images.

Problem

  • The ReaR error occurrs during grub2-mkimage of bootx64.efi.

Solution

  • To solve the problem, install the grub2-efi-x64-modules package.

SLES-related issues

SM_SSH does not work on SLES11 recovery image

Solution

  • In this case, execute
  • mount -t tmpfs none /dev/shm/ -o rw,nosuid,nodev,noexec
    

    before starting the recovery process.

Client is unreachable after booting the rescue image

Problem

  • After booting the rescue image the client is not reachable.

Solution

  • Start the client manually using the following command on the rescue command line:
  • sh /etc/scripts/system-setup.d/59-start-sesam-client.sh
    

EFI bootable image cannot be created on SLES11

Problem

  • EFI bootable image of GRUB2 cannot be created on SLES11.

Cause

  • SEP sesam v. 4.4.3.64 Grolar is the last version that supports SLES11 with UEFI.

Solution

  • To continue using SLES with UEFI, you should not upgrade to a later version of SEP sesam.

Installing ebiso for creating UEFI aware ISO images

In order to be able to create an UEFI/EFI bootable ISO image the, the additional tool ebiso has to be installed on the client system. This package is not part of a regular SLES12/SLES15 installation and can be downloaded at the following URL:

http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/SLE_12/x86_64/

or

http://download.sep.de/utils/bsr-linux/

For other Linux distributions contact SEP support at support@sep.de for assistance.

Install ebiso as follows:

rpm -i ebiso-<version>.rpm

Note that in ReaR v. < 1.19, the generated ISO image mount migt be too small for storing all needed information and need to be adjusted.

In this case, under

/var/opt/sesam/var/lib/rear/usr/share/rear/lib/uefi-functions.sh (line 64)

change

(shim.efi|elilo.efi) size=128000 ;;

to

(shim.efi|elilo.efi) size=228000 ;;

See also

SEP sesam Disaster Recovery