Archive:Disaster Recovery for Linux 3.0: Difference between revisions

From SEPsesam
No edit summary
(29 intermediate revisions by 6 users not shown)
Line 6: Line 6:


=== Requirements ===
=== Requirements ===
* SEP Sesam client >= 4.2.2
* SEP Sesam Client >= 4.2.2
* SEP Sesam Server >= 4.2.2
* SEP Sesam Server >= 4.2.2. Check [[Special:MyLanguage/Hardware_requirements|Hardware requirements]] for [[Special:MyLanguage/SEP_sesam_Glossary#SEP_sesam_Server|SEP sesam Server]] or [[Special:MyLanguage/SEP_sesam_Glossary#RDS|RDS]].
* Enough free space in '''/tmp/''', '''/boot/''' and '''/var/opt/sesam/''' to keep the temporary ISO image (ca. 100 MB, depending on the system)!
* Enough free space in '''/tmp/''', '''/boot/''' and '''/var/opt/sesam/''' to keep the temporary ISO image (ca. 100 MB, depending on the system)!


Line 14: Line 14:
See the following list in our support matrix:
See the following list in our support matrix:


[[SEP_sesam_OS_and_Database_Support_Matrix#SEP sesam Bare Metal Recovery Linux|SEP REAR Support Matrix]]
[[SEP_sesam_OS_and_Database_Support_Matrix#SEP_sesam_Bare_Metal_Recovery_Linux|SEP REAR Support Matrix]]


==== Package requirements ====
==== Package requirements ====
Line 25: Line 25:
* Backup of Sesam Server and Sesam RDS is currently not supported
* Backup of Sesam Server and Sesam RDS is currently not supported
* Backup of Systems running on Citrix XEN Virtual Machines is currently not supported
* Backup of Systems running on Citrix XEN Virtual Machines is currently not supported
* Backup of Systems using UEFI is currently not supported


==== Dissimilar Hardware ====
==== Dissimilar Hardware ====


In case you want to provide dissimilar hardware scenarios in your environment, please make sure to test this behavior beforehand. As SEP Sesam for REAR is generally able to restore to dissimiliar hardware there can always be certain limitations. As SEP cannot guarantee the dissimiliar hardware restore to work on every platform we do not give general support for it. In case you want to restore to dissimilar hardware, please test the restore szenario carefully to see if there are any limitations.
In case you want to provide dissimilar hardware scenarios in your environment, please make sure to test this behavior beforehand. As SEP Sesam for REAR is generally able to restore to dissimiliar hardware there can always be certain limitations. As SEP cannot guarantee the dissimiliar hardware restore to work on every platform we do not give general support for it. In case you want to restore to dissimilar hardware, please test the restore szenario carefully to see if there are any limitations.
==== (Multipath-)SAN Volumes / SAP HANA ====
By default no layout of (Multipath-)SAN volumes is saved, this means that during recovery there will be no way to recover partitions of this kind. If you are in need to backup in a multipath san setup, or SAP HANA installations, please get in touch with support@sep.de


== Backup ==
== Backup ==
Line 36: Line 41:


<!-- [[File:rear2.jpg]] -->
<!-- [[File:rear2.jpg]] -->
[[File:BSR-Linux3-Funtionality.jpg]]
[[File:BSR-Linux3-Funtionality.jpg|link=]]




Line 52: Line 57:




[[File:rear_backup_1.jpg]]
[[File:rear_backup_1.jpg|link=]]




[[File:rear_backup_2.jpg]]
[[File:rear_backup_2.jpg|link=]]


==== Special task options ====
==== Special task options ====
Line 62: Line 67:


  -a nodelete : do not delete generated iso image after backup  
  -a nodelete : do not delete generated iso image after backup  
-a raw : Do backup file in plain ISO Image format. This allows to burn image to bootable media directly from the backup file in the DataStore
  -v 3 : create REAR Logfile with debugging output (stored in: /var/opt/sesam/var/lib/rear/var/log/rear/)
  -v 3 : create REAR Logfile with debugging output (stored in: /var/opt/sesam/var/lib/rear/var/log/rear/)


==== Filesystem backup ====
==== Filesystem backup ====
The second backup task is needed to recover a system completely is a full backup of the clients filesystem (Backup type: "Path Backup" Source: "all"):
The second backup task is needed to recover a system completely is a full backup of the clients filesystem ([[Special:MyLanguage/SEP_sesam_Glossary#task_type|Task type]]: ''Path Backup''; Source: ''all''):




[[File:rear_backup_3.jpg]]
[[File:rear_backup_3.jpg|link=]]




Line 81: Line 87:


<!--[[File:rear_restore.jpg]]-->
<!--[[File:rear_restore.jpg]]-->
[[File:BSR-Linux3-Recovery.jpg]]
[[File:BSR-Linux3-Recovery.jpg|link=]]




Line 87: Line 93:


# Restore the ISO image to a different Linux machine, mount it to the recovery host or create a CDROM and use it as boot medium
# Restore the ISO image to a different Linux machine, mount it to the recovery host or create a CDROM and use it as boot medium
# The administrator boots the recovery console either via the created CDROM Image (2.1) or - if possible - via the systems booloader (2.2)
# The administrator boots the recovery console either via the created CDROM Image (2.1) or - if possible - via the systems bootloader (2.2)
# The administrator recovers the systems partitions and filesystem configuration via the recovery console
# The administrator recovers the systems partitions and filesystem configuration via the recovery console
# A restore of the complete filesystem backup of the client is performed to the newly created partition
# A restore of the complete filesystem backup of the client is performed to the newly created partition
Line 104: Line 110:




[[File:rear_restore_1.jpg]]
[[File:rear_restore_1.jpg|link=]]




Line 112: Line 118:




[[File:rear_restore_2.jpg]]
[[File:rear_restore_2.jpg|link=]]




Line 119: Line 125:




[[File:rear_restore_3.jpg]]
[[File:rear_restore_3.jpg|link=]]




Line 126: Line 132:




[[File:rear_restore_4.jpg]]
[[File:rear_restore_4.jpg|link=]]




Line 132: Line 138:




[[File:rear_restore_5.jpg]]
[[File:rear_restore_5.jpg|link=]]




Line 138: Line 144:




[[File:rear_restore_6.jpg]]
[[File:rear_restore_6.jpg|link=]]




[[File:rear_restore_7.jpg]]
[[File:rear_restore_7.jpg|link=]]




Line 147: Line 153:




[[File:rear_restore_8.jpg]]
[[File:rear_restore_8.jpg|link=]]




Line 154: Line 160:




[[File:rear_restore2_1.jpg]]
[[File:rear_restore2_1.jpg|link=]]




[[File:rear_restore2_2.jpg]]
[[File:rear_restore2_2.jpg|link=]]




Line 165: Line 171:




[[File:rear_restore2_3.jpg]]
[[File:rear_restore2_3.jpg|link=]]




Line 172: Line 178:
==== Remote login via SSH ====
==== Remote login via SSH ====


The recovery process can also be initiated remotely, means that the booted system image is reachable via SSH. Username for login is 'root' and the password is either empty or 'recover'.
The recovery process can also be initiated remotely, means that the booted system image is reachable via SSH. Username for login is 'root' and the password is either empty or 'recover'. With version >= 4.4.3 the password can be configured via special backup option:
 
-a SSH_ROOT_PASSWORD=<password>
 
in the backup task options.


== Troubleshooting ==
== Troubleshooting ==


=== Known Issues ===
=== Known Issues ===
==== Error Executing REAR ====
;Problem
SEP sesam Client package was successfully installed and path backups are working. If you have problems executing a rear backup check the backup log if there are some missing dependencies.
<pre>
There are binaries or libraries in the ReaR recovery system that need additional libraries
/opt/sesam/bin/sesam/libvirtmod.so requires additional libraries (fatal error)
libvirt.so.0 => not found
</pre>
=> In this case the package ''libvirt-client'' is missing.
* On RHEL/CentOS 7 install it with the following command: <tt>yum install libvirt-client</tt>
* On SLES 12/15 install it with the following command: <tt>zypper install libvirt-client</tt>


==== REAR image hangs during bootup ====
==== REAR image hangs during bootup ====
Line 182: Line 204:
In case the system hangs during bootup like shown in the following image:
In case the system hangs during bootup like shown in the following image:


[[File: rear-hang.jpg]]
[[File: rear-hang.jpg|link=]]


try booting the system with the ACPI=OFF option (this option can be specified on the command line in the boot menu prompt, after the options BACKUP=SESAM OUTPUT=ISO)
try booting the system with the ACPI=OFF option (this option can be specified on the command line in the boot menu prompt, after the options BACKUP=SESAM OUTPUT=ISO)
Line 200: Line 222:


* Press a key when prompted by the boot loader (GRUB):
* Press a key when prompted by the boot loader (GRUB):
[[File: rhelcentos_grub1.jpg]]
[[File: rhelcentos_grub1.jpg|link=]]




* Select the appropriate boot loader entry:
* Select the appropriate boot loader entry:
[[File: rhelcentos_grub2.jpg]]
[[File: rhelcentos_grub2.jpg|link=]]




* Press 'e' to edit the commands for the selected entry:
* Press 'e' to edit the commands for the selected entry:
[[File: rhelcentos_grub3.jpg]]
[[File: rhelcentos_grub3.jpg|link=]]




* Add 'selinux=0' to the commands:
* Add 'selinux=0' to the commands:
[[File: rhelcentos_grub4.jpg]]
[[File: rhelcentos_grub4.jpg|link=]]




Line 225: Line 247:
==== RHEL7  ====
==== RHEL7  ====


==== backup fails with ERROR: The LSB package is not installed ====  
===== backup fails with ERROR: The LSB package is not installed =====


If the backup fails with:
If the backup fails with:
Line 233: Line 255:
Please install the lsb package:
Please install the lsb package:


  yum install redhat-lsb-core
  yum install redhat-lsb-core mkisofs syslinux
 
===== backup fails with: ERROR: Cannot find required programs: mingetty =====
 
See the following documentation:
 
https://github.com/rear/rear/issues/348
 
Please edit:
 
/var/opt/sesam/var/lib/rear/usr/share/rear/conf/default.conf
 
in and in the line:
 
# required programs. Same as above, but if they are missing, we abort.
REQUIRED_PROGS=(
"$SCRIPT_FILE"
 
remove the line:
 
mingetty
 
and try the backup again.
 
==== Debian9  ====
===== Client won't start in recovery image =====
 
The Sesam client will not automatically start on Debian9 based Recovery images. The client can be started manually via:
 
/opt/sesam/bin/sesam/sm_main start
 
==== SLES ====
===== UEFI =====
 
====== Install EBISO for creating UEFI aware ISO images ======
 
In order to be able to create an UEFI/EFI bootable iso image the additional tool "ebiso" has to be installed on the client system. This Package is not part of a regular SLES12 installation and can be downloaded at the following URL:
 
http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/SLE_12/x86_64/
 
For other Linux distributions please contact support@sep.de
 
Installation via:
 
rpm -i ebiso-<version>.rpm
 
====== Additional configuration ======
 
 
In some cases the generated ISO Image mount will be too small for storing all needed information and has to be adjusted, this is will be fixed in future REAR verisons (1.19):
 
'''/var/opt/sesam/var/lib/rear/usr/share/rear/lib/uefi-functions.sh''' (line64):
 
change:
 
(shim.efi|elilo.efi) size=128000 ;;
 
to:
 
(shim.efi|elilo.efi) size=228000 ;;
 
===== restore: client is unreachable =====
 
after booting the rescue image the client is not reachable, please start the client manually using the following command on the rescue command line:
 
sh /etc/scripts/system-setup.d/59-start-sesam-client.sh


== Further documentation ==
== Further documentation ==
* For further information about ''Relax and Recover'' [http://relax-and-recover.org/ see here]
* For further information about ''Relax and Recover'' [http://relax-and-recover.org/ see here]

Revision as of 16:53, 2 May 2019

Template:Copyright SEP AG en


Introduction

This document includes information about SEP Sesam BSR Linux with REAR integration. Relax-and-Recover (REAR) (http://relax-and-recover.org/) is a setup-and-forget Linux bare metal disaster recovery solution based on GPL licence. This solution has been integrated into a SEP sesam backup client installation.

Requirements

  • SEP Sesam Client >= 4.2.2
  • SEP Sesam Server >= 4.2.2. Check Hardware requirements for SEP sesam Server or RDS.
  • Enough free space in /tmp/, /boot/ and /var/opt/sesam/ to keep the temporary ISO image (ca. 100 MB, depending on the system)!

Supported distributions

See the following list in our support matrix:

SEP REAR Support Matrix

Package requirements

There are some requirements which packages have to be installed, that a ReaR-based backup can be performed successfully.

  • RHEL 6
    • syslinux

Limitations

  • Backup of Sesam Server and Sesam RDS is currently not supported
  • Backup of Systems running on Citrix XEN Virtual Machines is currently not supported
  • Backup of Systems using UEFI is currently not supported

Dissimilar Hardware

In case you want to provide dissimilar hardware scenarios in your environment, please make sure to test this behavior beforehand. As SEP Sesam for REAR is generally able to restore to dissimiliar hardware there can always be certain limitations. As SEP cannot guarantee the dissimiliar hardware restore to work on every platform we do not give general support for it. In case you want to restore to dissimilar hardware, please test the restore szenario carefully to see if there are any limitations.

(Multipath-)SAN Volumes / SAP HANA

By default no layout of (Multipath-)SAN volumes is saved, this means that during recovery there will be no way to recover partitions of this kind. If you are in need to backup in a multipath san setup, or SAP HANA installations, please get in touch with support@sep.de

Backup

The SEP Sesam Disaster Recovery allways includes a special Linux BSR backup task and a complete file system backup of the clients. This documentation explains what is happening during the backup and how the backup tasks should be configured.

Functionality

BSR-Linux3-Funtionality.jpg


During backup the following steps are performed:

  1. The backup task creates a bootable CDROM or DVD image from the running system, including the SEP Sesam client
  2. The backup task creates a boot option in the systems bootloader which can be booted in case that the original boot config fails
  3. The created iso image is transferred to the backup server and backed up by a regular file backup
  4. A regular file backup is performed with a second task

Creating the backup tasks

Disaster Recovery Task

First step is to create a backup task for the Linux BSR console, this happens by choosing the Linux BSR backup option while browsing the client. This backup task will create the bootable CD / DVD image and the appropriate new bootloader entries on the client. This backup task should always be performed before the regular file backup.


Rear backup 1.jpg


Rear backup 2.jpg

Special task options

The following special options can be passed to the backup job "additional save options":

-a nodelete : do not delete generated iso image after backup 
-a raw : Do backup file in plain ISO Image format. This allows to burn image to bootable media directly from the backup file in the DataStore
-v 3 : create REAR Logfile with debugging output (stored in: /var/opt/sesam/var/lib/rear/var/log/rear/)

Filesystem backup

The second backup task is needed to recover a system completely is a full backup of the clients filesystem (Task type: Path Backup; Source: all):


Rear backup 3.jpg


Creating the bootable restore disk image

After the Disaster Recovery Task has completed, a restore of it has to be performed to gain the bootable .iso file containing the recovery environment. Therefor, a restore of the BSR task (rear-test_BSR in the pictured example) is done using the standard options.

When the restore task has completed successfully, the iso file stored in /var/opt/sesam/var/lib/rear/var/lib/rear/output/rear-[hostname].iso on the client can now be used to create a bootable media for the restore.

Restore

Functionality

BSR-Linux3-Recovery.jpg


During recovery the following steps are performed:

  1. Restore the ISO image to a different Linux machine, mount it to the recovery host or create a CDROM and use it as boot medium
  2. The administrator boots the recovery console either via the created CDROM Image (2.1) or - if possible - via the systems bootloader (2.2)
  3. The administrator recovers the systems partitions and filesystem configuration via the recovery console
  4. A restore of the complete filesystem backup of the client is performed to the newly created partition

After the steps have been performed successfully the system is rebooted

Performing a restore

Restoring the CDROM Image to a filesystem

In case the system is not bootable via the recovery console anymore (in case of a complete filesystem crash) the system must be booted via the recovery console CDROM. The first step is restoring the created ISO image to a random system and burning it to a media of your choice.

Booting the System from external media like USB/CDROM (or on-disk recovery console)

Bootmenu

After the media has been inserted the following boot menu will appear:


Rear restore 1.jpg


Networking

During bootup the system will provide you with a selection of network interfaces. Choose the right network interface which is needed to ensure connectivity with your backup server. The IP configuration of the original system will be used:


Rear restore 2.jpg


Login to the disaster recovery console

After the system has finished booting you can login to the disaster recovery console with the username "root" and no password. The disaster recovery console is also reachable via SSH, in case you have to login remotely. The recovery process can be started with the command "rear recover" just as shown in this screenshot:


Rear restore 3.jpg


Restoring disk information

The disk restore wizard will suggest a local disk to be partitioned, select the appropriate disk to be used:


Rear restore 4.jpg


Confirm the disk layout configuration:


Rear restore 5.jpg


Start the recovery and let the recovery console create the filesystems and partitions:


Rear restore 6.jpg


Rear restore 7.jpg


After the recovery check wether if all filesystem have been created, in the screenshot above you can see that there is an newly created mountpoint "/mnt/local". This will be the restore target for the next step: restoring the filesystem data.


Rear restore 8.jpg


Restoring the filesystem backup

After the partitions and filesystems have been created the next step is to restore the files of the system to the recovery console.


Rear restore2 1.jpg


Rear restore2 2.jpg


Restore target must be set to /mnt/local/

The restore target must be changed from the default value "all" to the new target "/mnt/local/" on the recovery console


Rear restore2 3.jpg


After the filesystem backup has been restored exit the disaster recovery console with the command "exit" and reboot the system.

Remote login via SSH

The recovery process can also be initiated remotely, means that the booted system image is reachable via SSH. Username for login is 'root' and the password is either empty or 'recover'. With version >= 4.4.3 the password can be configured via special backup option:

-a SSH_ROOT_PASSWORD=<password>

in the backup task options.

Troubleshooting

Known Issues

Error Executing REAR

Problem

SEP sesam Client package was successfully installed and path backups are working. If you have problems executing a rear backup check the backup log if there are some missing dependencies.

There are binaries or libraries in the ReaR recovery system that need additional libraries
/opt/sesam/bin/sesam/libvirtmod.so requires additional libraries (fatal error)
	libvirt.so.0 => not found

=> In this case the package libvirt-client is missing.

  • On RHEL/CentOS 7 install it with the following command: yum install libvirt-client
  • On SLES 12/15 install it with the following command: zypper install libvirt-client

REAR image hangs during bootup

In case the system hangs during bootup like shown in the following image:

Rear-hang.jpg

try booting the system with the ACPI=OFF option (this option can be specified on the command line in the boot menu prompt, after the options BACKUP=SESAM OUTPUT=ISO)

RHEL6/CENTOS6

Client won't start in recovery image

The Sesam client will not automatically start on RHEL6 based Recovery images as the file "/etc/init.d/functions" is missing within the recovery image. The client can be started manually via:

/opt/sesam/bin/sesam/sm_main start
Recovered system wont boot because of missing "libblkid.so.1"

If the recovered system won't boot, this is most probably caused by SELinux which is activated by default. Especially on RedHat Enterprise Linux 6 or CentOS 6 systems, follow these steps after rebooting from the REAR recover:

  • Press a key when prompted by the boot loader (GRUB):

Rhelcentos grub1.jpg


  • Select the appropriate boot loader entry:

Rhelcentos grub2.jpg


  • Press 'e' to edit the commands for the selected entry:

Rhelcentos grub3.jpg


  • Add 'selinux=0' to the commands:

Rhelcentos grub4.jpg


  • Press 'Enter' to confirm the changes and 'b' to boot up the machine with SELinux disabled.

When having access to the system, change the option SELINUX of /etc/selinux/config to the following:

SELINUX=permissive

Afterwards, reboot the system and feel free to set the SELINUX value back to 'enforcing' if needed.


RHEL7

backup fails with ERROR: The LSB package is not installed

If the backup fails with:

ERROR: The LSB package is not installed.

Please install the lsb package:

yum install redhat-lsb-core mkisofs syslinux
backup fails with: ERROR: Cannot find required programs: mingetty

See the following documentation:

https://github.com/rear/rear/issues/348

Please edit:

/var/opt/sesam/var/lib/rear/usr/share/rear/conf/default.conf

in and in the line:

# required programs. Same as above, but if they are missing, we abort.
REQUIRED_PROGS=(
"$SCRIPT_FILE"

remove the line:

mingetty

and try the backup again.

Debian9

Client won't start in recovery image

The Sesam client will not automatically start on Debian9 based Recovery images. The client can be started manually via:

/opt/sesam/bin/sesam/sm_main start

SLES

UEFI
Install EBISO for creating UEFI aware ISO images

In order to be able to create an UEFI/EFI bootable iso image the additional tool "ebiso" has to be installed on the client system. This Package is not part of a regular SLES12 installation and can be downloaded at the following URL:

http://download.opensuse.org/repositories/Archiving:/Backup:/Rear/SLE_12/x86_64/

For other Linux distributions please contact support@sep.de

Installation via:

rpm -i ebiso-<version>.rpm
Additional configuration

In some cases the generated ISO Image mount will be too small for storing all needed information and has to be adjusted, this is will be fixed in future REAR verisons (1.19):

/var/opt/sesam/var/lib/rear/usr/share/rear/lib/uefi-functions.sh (line64):

change:

(shim.efi|elilo.efi) size=128000 ;;

to:

(shim.efi|elilo.efi) size=228000 ;;
restore: client is unreachable

after booting the rescue image the client is not reachable, please start the client manually using the following command on the rescue command line:

sh /etc/scripts/system-setup.d/59-start-sesam-client.sh

Further documentation

  • For further information about Relax and Recover see here