5 1 0:Backup to S3 Cloud Storage

From SEPsesam

Welcome to the latest SEP sesam documentation version 5.1.0 Apollon. For previous documentation version(s), check documentation archive.


SEP sesam v. 5.0.0 Jaglion has introduced a new generation Si3 data store. It enables you to back up your data directly to the S3 (Simple Storage Solution) cloud storage and restore the items you want directly from there. After an initial full backup of your virtual and physical environment, you can use any backup level (including differential and incremental backups) to back up only new data to the cloud.

Si3 offers significantly improved performance for backup, restore and migration, resulting in improved performance, scaling and resource savings. For a comparison of the two deduplication stores, see the section Comparison of Si3 V1 and Si3 below.

Powerful restore

The new Si3 can detect duplicate data fragments to streamline the restore process. Use the Web Restore Assistant or GUI Restore Wizard to instantly restore your data from backups. Restoring a single file is easy as you can use the search function to find the desired backup and start the restore process to the to the original or another location.

Si3 uses S3 in the same way as local storage, allowing you to use your S3 cloud storage to securely store and retrieve your business data anytime, anywhere.

Multiple media pools

Si3 lets you create multiple media pools to provide scalability and granularity of backup jobs. You can create separate media pools, e.g., for daily differential and incremental and weekly full backup jobs, or for migration and replication, and you can set a different retention time for each media pool.

SEP sesam support for S3-compatible cloud and blob storage

With Si3, you can back up your data directly to the S3 cloud and (from Jaglion V2) to Microsoft Azure. As S3 is an open API standard and AWS Simple Storage Service is a sample implementation of the standard, SEP sesam Si3 can also be used with other S3-compatible cloud implementations. The configuration and management of Si3 in an S3-compatible cloud implementation is similar to the example shown in this article and must follow the same process and rules provided for using Si3 with S3.

In Azure, read access carries higher costs. Tasks such as housekeeping, consistency checks, and restores will incur higher expenses. Consider this when planning your operations.

Note, however, that some S3-compatible cloud environments have not yet been validated by SEP and may not work in some cases. You are solely responsible for the use of SEP sesam in a non-S3 compatible cloud environment and agree that SEP shall not be responsible for it. For the list of supported object storage, see the support matrix.

Updating Si3 on S3 from to the new version

If you use Si3 on S3 and update from to the new version, the structure of the existing stores will change as the structure of Si3 on S3 is automatically recreated (this includes recreating the index after the renaming). Example:

  • The S3 bucket is called seps3, the Si3 deduplication store name is newSi3. The S3 structure with version of Si3 is: seps3/pages; seps3/pages-trash; seps3/objects-trash.
  • When updating to the next version of Si3, the structure changes to: seps3/newSi3/pages; seps3/newSi3/pages-trash; seps3/newSi3/objects-trash. During this renaming, the Si3 service is not available.

Configuring secure communication with self-signed certificates

Si3 deduplication store enables using secure network communication protocols. Public S3 providers typically use signed certificates. However, if the S3 provider does not use a public certificate issued by a certification authority (CA), or if you use self-signed certificates, you must add this certificate to the Java keystore. Import the self-signed certificate on the SEP sesam Server or the RDS/RTS where Si3 is running to configure secure communication and establish an HTTPS/TLS connection to the S3 cloud storage.

You can use the following command:

keytool -import -trustcacerts -keystore <keystore_path> -storepass <keystore_password> -noprompt -alias <alias_name> -file <certificate_file_path>
<keystore_path>: the path to the Java keystore file
<keystore_password>: the password for the Java keystore
<alias_name>: an alias for the certificate (for example, hostname or IP address)
<certificate_file_path>: the path to the certificate file that you want to import
  • on Linux:
keytool -import -trustcacerts -keystore /var/lib/ca-certificates/java-cacerts -storepass changeit  -noprompt -alias -file /homes/users/rs/copy/public.crt
  • on Windows:
keytool -import -trustcacerts -keystore "C:\Program Files\ojdkbuild\java-11-openjdk-11.0.15-1\lib\security\cacerts" -storepass changeit -noprompt -alias sesamserverix.sep.de -file C:\rs\cert_mini_java\public.crt

Configuration of Si3 S3 store

SEP sesam enables you to back up your data directly to S3 cloud storage using Si3 deduplication store. This procedure contains only the basic steps. For details on Amazon S3 storage configuration, see the Amazon Simple Storage Service Documentation.

  1. Create a bucket
  2. Create a new backup user
  3. Configure the Si3 deduplication store
  4. Configure media pools
  5. Configure backups

Create a bucket

Sign-up for S3 and create a bucket (container) where every object in S3 is stored. The term bucket is used to describe the container for backup data.

  1. Sign in to the AWS Management Console and open the Amazon S3 console at https://console.aws.amazon.com/s3/
  2. Select Create bucket and enter all required information. Follow the Amazon Simple Storage Service instructions: Creating a Bucket.

Create a new backup user

On S3, configure an AWS identity and IAM (Access Management user <user_name> with administrative user rights. Sign in to the IAM console as the account owner by selecting Root user and providing your AWS account email address.
Follow the recommendation and use your root user credentials only to create your IAM admin user. For details, see the AWS Identity and Access Management documentation: Creating an Administrator IAM User and Group.

AWS access keys consist of an access key ID and a secret access key. Both keys are required to authenticate access. These credentials are also required to create an Si3 data store for S3.

The S3 credentials are stored encrypted (not in plain text) in the ini file.

Configure the Si3 deduplication store

  1. In the Main selection -> Components, click Data Stores to display the data store contents frame.
  2. From the Data Stores menu, select New Data Store. A New Data Store dialog appears.
  3. Under Data store properties, enter a meaningful name for the Si3 deduplication store in the Name field, e.g., Si3-NG-S3. Entering the name also creates the name of the drive group for your Si3 deduplication store in the Create new drive group field.
  4. From the Store type drop-down list, select SEP Si3 NG Deduplication Store.
  5. Under Drive parameter, leave the options Create drive and Create second drive checked. The predefined value for the drive is automatically entered in the Drive number field.
  6. By using the additional dedicated drive for the restore, you can perform a backup on the first drive and restore your data from the second drive simultaneously. You can also add a third drive for migration. (For details, see Drive access mode.)
  7. The name in Create new drive group is already created. You can change it by simply entering a new name.
  8. The predefined number of channels is already available in the Max. channels drop-down list. The number of available channels depends on your SEP sesam Server package. For details on licensing, see Licensing.
  9. From the Device server drop-down list, select the device server for your data store.
  10. In the Path field, enter the location or use the Browse button to select a directory on the local disk (as for local storage). This directory is only used to store metadata and temporary files for intermediate storage until the data is uploaded to S3. Ensure that there is sufficient disk-space available for this local storage, e.g., for 50 TiB in S3 or more, make sure it has 20 GiB free space. Click OK.
    If you use the Browse button, the New Data Store information window appears with predefined recommended values for the size of your Si3 deduplication store, based on the size of the previously selected local disk (Path).
    Important: Change these values manually under the Size properties, depending on how much storage space you want to use on S3.
    • Capacity: Set the size (in GB/GiB) of the storage for backups.
    • High watermark: The HWM defines the upper value for the used storage space. When this value is reached, the status of a datastore changes from OK to Warning, but backups continue to be performed. Make sure that you provide enough storage space for your backed up data.

    Click OK. You can also change the size of your Si3 deduplication store later under Size properties (see Size properties).

    Since S3 offers unlimited scalability and there is no official limit to the amount of data and number of objects you can store in an S3 bucket, you should set the capacity of Si3 on S3 according to your needs. If you know how much capacity you need, you can optimize the cost of your cloud services.

  11. Switch to the Storage Backend tab and select Storage type: Amazon S3 or compatible. Then click New to create a new credential set for S3. You will need to enter your previously configured credentials, see section Creating a new backup user above.
    • Credential set: give your credential set a name.
    • Endpoint URL: enter the endpoint URL for your S3 storage account.
    • In the Access key field, type the AWS access key ID (for example, AKIAIOSFODNN7EXAMPLE).
    • In the Secret key field, type the AWS secret access key (for example, wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY).
    • In the Verify secret key field, type the AWS secret access key again.
    • In the Bucket name field, enter the name of your bucket.

  12. If you want to create a media pool for your Si3 deduplication store immediately, click Create Media Pool. Otherwise, click OK to configure your Si3 deduplication store.

Configuring media pools for Si3

  1. From Main Selection -> Media Pools, click New Media Pool. The New Media Pool window is displayed.
  2. In the Media Pool window, specify the required fields:

    • Name: Enter a name for a media pool.
    • Description: Optionally, add a description of the pool.
    • Drive group: From the drop-down list of all available drive groups, select the relevant drive group to which a media pool will be attached. For details on drive groups, see Drives.
    • Retention time [days]: Specify the retention time for the media pool. The retention period begins with the date a saveset is written to the media (at the end time of the first backup) and thus defines the expiry date of the saveset - EOL. When the protection (EOL) expires, SEP sesam can use the media for backups again. For details, see What is retention time and EOL.

You can repeat the procedure and create more media pools for Si3 according to your needs.

Configuring backup

After you have configured an Si3 deduplication store and one or more media pools, proceed as follows:

  1. Create a backup task to back up to S3.
  2. Configure a schedule as described in Creating a Schedule.

Once you have configured a task and schedule, create a backup event as follows.

Creating a backup event

By creating a backup event, you select the backup level, set event priority, and specify where to back up your data. You can create an event for a specific task or for a task group. The latter enables you to trigger all the tasks in the task group with a single event.

  1. From Main Selection -> Scheduling -> Schedules, right-click the schedule you created earlier and click New Backup Event.
  2. Under the Sequence control, set the Priority of your backup event. SEPuler always executes the events with the highest priority first. The default priority level is 1, which is the lowest priority (the highest priority is 99). The only exception are schedules with priority 0, which override all other priorities and are always executed. For details, see Event priority. You can also enable the Blocking date. This option should be used together with the high priority for special events. When this option is enabled, events of the same type but with a lower priority are blocked, so that the backup is executed even if other backups are scheduled for the same time.
  3. Under Object, select the task (or task group) you configured earlier and with which you want to link this event.
  4. Under Parameter, specify the Backup level.
  5. From the Media pool drop-down list, select the media pool you created for the Si3 deduplication store. The data is backed up to this pool. You can also activate the option SEP Si3 source-side deduplication.

    • Optionally, specify the drive number of the drive to be used to write the data. Typically, you use this option if you have configured additional drives and want to assign a specific drive exclusively for backup.
  6. In the Follow up field, you can configure events (e.g., migration) that are triggered on the SEP sesam Server as soon as the first event (e.g., backup) is completed. For details, see Follow-up events.
  7. Tip
    You can set a follow-up migration task by selecting the previously configured migration task from the Migration task drop-down list.

Monitoring backups

You can view the status of your backup jobs in the GUI (Monitoring -> Last Backup State or Job State -> Backups) or SEP sesam Web UI. The backup status overview provides detailed information about the last run of backup jobs, including the task name, start and stop time of the last backup, backup level, data size, throughput, assigned media pool, etc.

Purging data on S3 storage

Data purging is the process of permanently deleting obsolete (EOL-free) savesets from regular data stores. It works in the same way in the S3 cloud. The purge is automatically triggered and performed until all EOL-free savesets are deleted.

Automatically purging data

Strategies for data purging are based on the nature of your business, as well as regulatory, legal and other requirements, and implemented with a defined data retention period. This is the period of time for which backup data is protected after it is written to the media, so that the savesets are preserved and available for restore. It is based on the media pool retention time you set when you created a media pool. SEP sesam provides automatic EOL (retention) management to ensure recoverability of the entire backup chain and protect against data loss, based on the backup chain dependencies. For more details, see What are backup chain dependencies.

When protection (EOL) expires, purging is done automatically and SEP sesam can use the media for backups again. For more details, see What happens when retention expires.

Events that trigger the data store purge are:

  • Manual start of the purge in the GUI
Manually purging data
  • You can manually adjust the EOL of your data or delete a saveset or backup. For details, see Changing Retention (EOL).
  • You can start the data store purge process in the GUI: Main Selection window -> Components -> Data Stores content pane -> option Purge. Running the purge manually deletes the obsolete (EOL-free) savesets.
  • Another way to free up storage space is to remove orphaned savesets from the data stores by using the Clean up option: Main Selection window -> Components -> Data Stores content pane -> option Clean up. This is useful if a data store seems to be inaccessible, its space is occupied, or SEP sesam space check shows non-sesam data.

Comparison of Si3 V1 and Si3

SEP sesam v. 5.0.0 Jaglion has introduced a new generation Si3 deduplication store. Si3 offers significantly higher performance for backup, restore and migration, as well as backup to S3 cloud and backup to Azure, the new immutable storage feature SiS, resulting in improved performance, scaling, and resource savings.

Function Si3 Si3 NG
Si3 backup YesY YesY
Si3 deduplication (source-side and target-side) YesY YesY
Si3 replication: local to remote store Notea YesY Si3 V1 to Si3 V1 YesY Si3 V1 to Si3; Si3 to Si3
Si3 replication: to S3 cloud YesY NoN (provides more powerful features for backing up directly to the cloud, see the next two lines)
Backup to S3 Cloud Storage NoN YesY
Backup to Azure Storage NoN YesY (as of Jaglion V2)
SiS (SEP Immutable Storage) NoN YesY (as of Jaglion V2)
Si3 restore YesY YesY
Si3 encryption YesY YesY (as of Jaglion V2)
Seeding Si3 deduplication store Noteb YesY NoN
Usage of tachometer YesY NoN

SEP sesam does not support a direct upgrade from the Si3 V1 to new Si3. However, to use the new Si3 you can:

  • Back up all data again to the newly configured Si3 deduplication store.
  • After configuring a new Si3, you can also create a replication job to replicate from the Si3 V1 to the Si3 store. Replication reads all the data from the source-side store on the source-side RDS and sends it to the target store using the source-side deduplication function. For details, see Replicating from Si3 V1 to Si3.
  • You can also configure a new Si3 and an old Si3 V1 in parallel on the same host by enabling the key enable_gui_allow_multi_dedup.

The Initial Seed feature does not work in v. 5.0.0 Jaglion, but you can use it in earlier SEP sesam versions.

The new immutable storage feature (introduced in Jaglion V2) is also based on Si3 store (set up on a dedicated Linux server). SiS is SEP Immutable Storage, based on the File Protection Service (FPS), which scans the file system and sets the immutable bit for all new objects. This means that all data stored in SiS is marked immutable at the time of storage. Even with full admin access to the SEP sesam backup server, attackers cannot delete, modify, or encrypt data stored on SiS. For details, see SEP Immutable Storage – SiS.

See also

Configuring Si3 Deduplication StoreEncrypting Si3 Deduplication StoreStandard Backup ProcedureRestore AssistantStandard Restore ProcedureLicensing

Copyright © SEP AG 1999-2024. All rights reserved.
Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.