Source:Analyzing SEP sesam Log Files

From SEPsesam
Revision as of 15:32, 19 July 2022 by Sta (talk | contribs)
Other languages:

Copyright © SEP AG 1999-2024. All rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Docs latest icon.png Welcome to the latest SEP sesam documentation version 4.4.3/5.0.0 Jaglion. For previous documentation version(s), check Documentation archive.


Overview

Analyzing SEP sesam log files is very useful to detect operations that have caused errors or malfunctions, for example, in the case of a failed backup. The log files are also used to track or audit changes to data, as described in Audit Logging.

SEP sesam creates two protocols or log files for each backup day: the status file and the day log. An error log is the subset of the entire day log, where only error messages are recorded. The log files can be printed or sent by email. The default location (main directory) for the log files is SESAM_VAR/log. You can check the backup logs (state, day or error) in the GUI (Main Selection -> Logging -> State/Day/Error Log).

You can also check your system logs online by using the Web UI ( ≥ 4.4.3 Beefalo V2). For details, see SEP sesam Web UI.

Log files creation order during a backup

When a scheduled backup is performed, log files are generated in a specific order. If you analyze a problem and find that the corresponding log files are missing from a certain point in time in the past, the cause of the problem is most likely positioned just before that point. An example of how a log file is generated is given below.

  1. The SEPuler creates the log file sm_sepul_event_xxx.log, for example sm_sepul_event_20181004.log.
  2. The queue manager writes the sm_qm_main_xxx.log, for example sm_qm_main_20180913.log.
  3. If the backup was able to start, a bck_*.log is created.
  4. When the backup starts, a backup .not log (notification) is created in the SEP sesam's SESAM_VAR/lis directory, e.g., smhg00_all- 20181004_001_SF20131004090011986@YlyxvqJCsHm.not.
  5. In case of optional media init, a sm_init_X_20180915.log is created. The X stands for the SEP sesam drive number.
  6. The information for monitoring drives and performance data is written in the sms log (sm_sms_watch_X_20181004.log).
  7. The files which are backed up are first written to a *.lis file (list of the backed up files and directories) and to sgm file (segment-file of the used segment markers on the used tapes) on the device server in the SESAM_VAR/work/smslis directory. Once the backup is finished, these files are copied from the device server to the SEP sesam SESAM_VAR/lis directory. This data is needed for a selective restore.

Course of action for log file analysis

The recommended course of action depends on the failure. For example, if a scheduled backup did not run or have failed, proceed as follows.

Check the backup log in the GUI/Web UI

From Main Selection -> Backups, double-click the relevant Failed backup and open the Main Log tab to check the backup log.

As of 4.4.3 Beefalo V2, you can also check your failed backups online by using new Web UI. For details, see SEP sesam Web UI.

Check if a media init error has occurred

If a medium init error occurred (bck_*.log), it is the cause of a failed backup.

Check if a backup has no log or has failed

If the backup does not yet have a log or has failed:

  1. Check the status and daily logs in the SESAM_VAR/prot directory for events and errors at that particular time.
  2. Check if a .not log exists in the SESAM_VAR/lis directory. If not, then the possible causes for the error are as follows:
    • Client is not accessible (DNS, ping).
    • There is no media available.
    • The backup did not start yet.
  3. The process logs are stored in the SESAM_VAR/log/lgc directory. The logs should be listed chronologically in the terminal, e.g., Linux ls -lart).
    • The bck_*.log is created by the program sm_backup; naming convention: bck_<job_name>_<save_set_ID>_<sesam_day>.log. Note that unlike most other logs, the bck_*.log must be read from the beginning to find the first error message that may reveal the cause of the failure.
      • Check the license.
      • Set time range: Check if the backup is within the set time frame.
      • Alive test: Check if the client is active and reachable.
      • CHECK_MEDIUM: Check the availability of the media.
        • iGET_PREPARED_MEDIA: Check the media pool. If msg=0 appears, there is no media available.
        • GET_BACKUP_MEDIUM: The sm_sms_interface is doing something on the tape (getlabel, init, etc.). For tasks started simultaneously, search for the largest backup file that contains the log files of the executed media init.
        • In the case of tape media, the following Options may be set as described in Configuring a Media Pool. They allow you to control which media to use for a backup. For example, for devices that load media in sequential order, or if you do not want an unattended backup to fail because the specified media are not available, you can use the empty media policy.
          • empty: If this option is selected and no EOL-free media are available in the requested media pool, SEP sesam will use any suitable media for the backup – empty media and media that are unrecognized by SEP sesam.
          • spare: If this option is selected and no EOL-free media are available in the requested media pool, media from the SPARE_ pool will be used for backup.
          • other: If this option is selected and no EOL-free media are available in the requested media pool, any EOL-free media from other media pools will be used.
        • Note: The EOL defines how long the backed up data on the media remains protected after the data is written to the media (see Managing EOL). When the protection expires, SEP sesam can re-use the media for backups. You can check the life cycle of a tape in the daily log; for details on media initialization, checked the *.sms log files in the <SESAM_VAR>/log/sms directory.
      • sm_notify is delivered.
      • Use the search pattern "Cmd= sbc" to jump directly to the command given in the log that calls up the backup.
      • If the backup is started successfully, the .not log is created in the SESAM_VAR/lis directory.
      • During an active backup all log information is appended to .not log.
    • If there are problems with the media init, the errors are recorded in the sm_init_<drive>_<sesam_day>.log. If a log includes the error: all media with eol restriction, then no media are available in the requested pool. There may be further attempts to get the backup media according to the specified media pool options (see above Options.
    • If there are problems with subsequent tapes after writing the backup data until the End Of Media (EOM) is reached, they will be written in the sm_sms_watch_<drive>_<sesam_day>.log. If this log includes error: all media with eol restriction, then no media are available in the requested pool.
    • To check the communication between the client and the SEP sesam Server either over SMSSH (default, via port 11322) or over CTRL (via port 11301), look for sm_sshd_<sesam_day>.log or sm_ctrld_<PID>.lgc in the SESAM_VAR>/log/lgc directory on the client. The logs on the client are generated when the SEP sesam Server performs a task on the client, for example:
      • Execution of a backup.
      • Execution of a command.
      • Browsing the GUI file wizard (when creating a task).
    • Whenever the SEPuler finds a task that has to be executed, the operation is recorded in (sm_sepul_event_<schedule_identifier>_<sesam_day>.log). With QUE_SUBMIT a job is put into a queue; sm_backup shows that the backup is being put into the queue.
    • On a SEP sesam Server, search the log sm_sms_watch_<drive>_<sesam_day>.log from bottom upwards to check drive information. This includes:
      • Monitoring a drive. This log is generated only on a SEP sesam Server (the corresponding watch logs for drives on RDS are also located on the SEP sesam Server).
      • Data throughput of a drive.
      • The search string "‘+++ EOM"‘ shows media changes related to the end of media (tape is full). The sm_sms_interface init command:
        • STATUS=SUCCESS: Successful media initialization.
        • STATUS=IO-ERROR: There is a problem with the media or a drive. If necessary, check the SESAM_VAR>/log/messages on Linux or the event log on Windows for any hardware problems. To confirm that there is a hardware problem, check the sms log in the <SESAM_VAR>/log/sms directory.
    • The log sm_sms_watch_0_<sesam_day>.log regularly displays the process status of the sm_main processes and the processes in the sm_qm_main queues, in addition to periodically checking the available space GET_FREE_SPACE_OF_DIR.