Archive:Disaster Recovery for Linux 3.0

From SEPsesam

Template:Copyright SEP AG en


Introduction

This document includes information about SEP Sesam BSR Linux with REAR integration. Relax-and-Recover (REAR) (http://relax-and-recover.org/) is a setup-and-forget Linux bare metal disaster recovery solution based on GPL licence. This solution has been integrated into a SEP sesam backup client installation.

Requirements

  • SEP Sesam Client >= 4.2.2
  • SEP Sesam Server >= 4.2.2. Check Hardware requirements for SEP sesam Server or RDS.
  • Enough free space in /tmp/, /boot/ and /var/opt/sesam/ to keep the temporary ISO image (ca. 100 MB, depending on the system)!

Supported distributions

See the following list in our support matrix:

SEP REAR Support Matrix

Package requirements

There are some requirements which packages have to be installed, that a ReaR-based backup can be performed successfully.

  • RHEL 6
    • syslinux

Limitations

  • Backup of Sesam Server and Sesam RDS is currently not supported
  • Backup of Systems running on Citrix XEN Virtual Machines is currently not supported

Dissimilar Hardware

In case you want to provide dissimilar hardware scenarios in your environment, please make sure to test this behavior beforehand. As SEP Sesam for REAR is generally able to restore to dissimiliar hardware there can always be certain limitations. As SEP cannot guarantee the dissimiliar hardware restore to work on every platform we do not give general support for it. In case you want to restore to dissimilar hardware, please test the restore szenario carefully to see if there are any limitations.

(Multipath-)SAN Volumes / SAP HANA

By default no layout of (Multipath-)SAN volumes is saved, this means that during recovery there will be no way to recover partitions of this kind. If you are in need to backup in a multipath san setup, or SAP HANA installations, please get in touch with support@sep.de

Backup

The SEP Sesam Disaster Recovery allways includes a special Linux BSR backup task and a complete file system backup of the clients. This documentation explains what is happening during the backup and how the backup tasks should be configured.

Functionality


During backup the following steps are performed:

  1. The backup task creates a bootable CDROM or DVD image from the running system, including the SEP Sesam client
  2. The backup task creates a boot option in the systems bootloader which can be booted in case that the original boot config fails
  3. The created iso image is transferred to the backup server and backed up by a regular file backup
  4. A regular file backup is performed with a second task

Creating the backup tasks

Disaster Recovery Task

First step is to create a backup task for the Linux BSR console, this happens by choosing the Linux BSR backup option while browsing the client. This backup task will create the bootable CD / DVD image and the appropriate new bootloader entries on the client. This backup task should always be performed before the regular file backup.



Special task options

The following special options can be passed to the backup job "additional save options": 

-a nodelete : do not delete generated iso image after backup 
-a raw : Do backup file in plain ISO Image format. This allows to burn image to bootable media directly from the backup file in the DataStore
-v 3 : create REAR Logfile with debugging output (stored in: /var/opt/sesam/var/lib/rear/var/log/rear/)

Filesystem backup

The second backup task is needed to recover a system completely is a full backup of the clients filesystem (Task type: Path Backup; Source: all):



Creating the bootable restore disk image

After the Disaster Recovery Task has completed, a restore of it has to be performed to gain the bootable .iso file containing the recovery environment. Therefor, a restore of the BSR task (rear-test_BSR in the pictured example) is done using the standard options.

When the restore task has completed successfully, the iso file stored in /var/opt/sesam/var/lib/rear/var/lib/rear/output/rear-[hostname].iso on the client can now be used to create a bootable media for the restore.

Restore

Functionality


During recovery the following steps are performed:

  1. Restore the ISO image to a different Linux machine, mount it to the recovery host or create a CDROM and use it as boot medium
  2. The administrator boots the recovery console either via the created CDROM Image (2.1) or - if possible - via the systems bootloader (2.2)
  3. The administrator recovers the systems partitions and filesystem configuration via the recovery console
  4. A restore of the complete filesystem backup of the client is performed to the newly created partition

After the steps have been performed successfully the system is rebooted

Performing a restore

Restoring the CDROM Image to a filesystem

In case the system is not bootable via the recovery console anymore (in case of a complete filesystem crash) the system must be booted via the recovery console CDROM. The first step is restoring the created ISO image to a random system and burning it to a media of your choice.

Booting the System from external media like USB/CDROM (or on-disk recovery console)

Bootmenu

After the media has been inserted the following boot menu will appear:



Networking

During bootup the system will provide you with a selection of network interfaces. Choose the right network interface which is needed to ensure connectivity with your backup server. The IP configuration of the original system will be used:



Login to the disaster recovery console

After the system has finished booting you can login to the disaster recovery console with the username "root" and no password. The disaster recovery console is also reachable via SSH, in case you have to login remotely. The recovery process can be started with the command "rear recover" just as shown in this screenshot:



Restoring disk information

The disk restore wizard will suggest a local disk to be partitioned, select the appropriate disk to be used:



Confirm the disk layout configuration:



Start the recovery and let the recovery console create the filesystems and partitions:




After the recovery check wether if all filesystem have been created, in the screenshot above you can see that there is an newly created mountpoint "/mnt/local". This will be the restore target for the next step: restoring the filesystem data.



Restoring the filesystem backup

After the partitions and filesystems have been created the next step is to restore the files of the system to the recovery console.




Restore target must be set to /mnt/local/

The restore target must be changed from the default value "all" to the new target "/mnt/local/" on the recovery console



After the filesystem backup has been restored exit the disaster recovery console with the command "exit" and reboot the system.

Remote login via SSH

The recovery process can also be initiated remotely, means that the booted system image is reachable via SSH. Username for login is 'root' and the password is either empty or 'recover'. With version >= 4.4.3 the password can be configured via special backup option: 

-a SSH_ROOT_PASSWORD=<password>

in the backup task options.

Troubleshooting

Known Issues

Error Executing REAR

Problem

SEP sesam Client package was successfully installed and path backups are working. If you have problems executing a rear backup check the backup log if there are some missing dependencies. 

There are binaries or libraries in the ReaR recovery system that need additional libraries
/opt/sesam/bin/sesam/libvirtmod.so requires additional libraries (fatal error)
	libvirt.so.0 => not found

=> In this case the package libvirt-client is missing.

  • On RHEL/CentOS 7 install it with the following command: yum install libvirt-client
  • On SLES 12/15 install it with the following command: zypper install libvirt-client

Error The workflow mkrescue is not supported in the ReaR rescue/recovery system

Delete the file /etc/rear-release

REAR image hangs during bootup

In case the system hangs during bootup like shown in the following image:

try booting the system with the ACPI=OFF option (this option can be specified on the command line in the boot menu prompt, after the options BACKUP=SESAM OUTPUT=ISO)

RHEL6/CENTOS6

Client won't start in recovery image

The Sesam client will not automatically start on RHEL6 based Recovery images as the file "/etc/init.d/functions" is missing within the recovery image. The client can be started manually via: 

/opt/sesam/bin/sesam/sm_main start
Recovered system wont boot because of missing "libblkid.so.1"

If the recovered system won't boot, this is most probably caused by SELinux which is activated by default. Especially on RedHat Enterprise Linux 6 or CentOS 6 systems, follow these steps after rebooting from the REAR recover:

  • Press a key when prompted by the boot loader (GRUB):


  • Select the appropriate boot loader entry:


  • Press 'e' to edit the commands for the selected entry:


  • Add 'selinux=0' to the commands:


  • Press 'Enter' to confirm the changes and 'b' to boot up the machine with SELinux disabled.

When having access to the system, change the option SELINUX of /etc/selinux/config to the following: 

SELINUX=permissive

Afterwards, reboot the system and feel free to set the SELINUX value back to 'enforcing' if needed.


RHEL7

backup fails with ERROR: The LSB package is not installed

If the backup fails with: 

ERROR: The LSB package is not installed.

Please install the lsb package: 

yum install redhat-lsb-core mkisofs syslinux
backup fails with: ERROR: Cannot find required programs: mingetty

See the following documentation: 

https://github.com/rear/rear/issues/348

Please edit: 

/var/opt/sesam/var/lib/rear/usr/share/rear/conf/default.conf

in and in the line: 

# required programs. Same as above, but if they are missing, we abort.
REQUIRED_PROGS=(
"$SCRIPT_FILE"

remove the line: 

mingetty

and try the backup again.

UEFI

In order to be able to create an UEFI/EFI bootable iso image the additional tool "ebiso" has to be installed on the client system.

rear error occurred during grub2-mkimage of bootx64.efi

Please install the grub2-efi-x64-modules package to fix this

SLES11: "Cannot make bootable EFI image of GRUB2"

On SLES11 with UEFI the last supported verison is included in sesam client 4.4.3.64, newer versions wont work with UEFI on SLES.

Debian9

Client won't start in recovery image

The Sesam client will not automatically start on Debian9 based Recovery images. The client can be started manually via: 

/opt/sesam/bin/sesam/sm_main start

SLES

SM_SSH not working on SLES11 recovery image

execute 

mount -t tmpfs none /dev/shm/ -o rw,nosuid,nodev,noexec

before starting the recovery process

UEFI
Install EBISO for creating UEFI aware ISO images

In order to be able to create an UEFI/EFI bootable iso image the additional tool "ebiso" has to be installed on the client system. This Package is not part of a regular SLES12/SLES15 installation and can be downloaded at the following URL: 

http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/SLE_12/x86_64/

or: 

http://download.sep.de/utils/bsr-linux/

For other Linux distributions please contact support@sep.de

Installation via: 

rpm -i ebiso-<version>.rpm
Additional configuration

In some cases the generated ISO Image mount will be too small for storing all needed information and has to be adjusted, this is will be fixed in future REAR verisons (1.19):

/var/opt/sesam/var/lib/rear/usr/share/rear/lib/uefi-functions.sh (line64):

change: 

(shim.efi|elilo.efi) size=128000 ;;

to: 

(shim.efi|elilo.efi) size=228000 ;;
restore: client is unreachable

after booting the rescue image the client is not reachable, please start the client manually using the following command on the rescue command line: 

sh /etc/scripts/system-setup.d/59-start-sesam-client.sh

Further documentation

  • For further information about Relax and Recover see here
Retrieved from "https://wiki.sep.de/wiki/index.php?title=Archive:Disaster_Recovery_for_Linux_3.0&oldid=317261"