Commit 02723457 authored by thomas.forbriger's avatar thomas.forbriger
Browse files

[TASK] (examples): add configuration example files

parent cde8e6d3
**/GR/BFO/?DF*
**/XM/BFO/H??*
**/XM/BFO/?L?*
**/XM/BFO/VM?\.D/
**/XE/BFO/H??*
**/XE/BFO/BA1*
**/XS/BFO/?X6*
**/XS/BFO/[HB][GD]?*
**/XS/BFO/VM?*
**/XG/BFO/?X?*
**/XG/BFO/HG?*
**/XT/
**/XW/
**/XX/
.lock
checksumfile.result*
this is <README>
============================================================================
Example configuration files for csback
======================================
Files
-----
BFO2LSDF.sl.exclude
exclude patterns for BFO2LSDF.sh
csback.csbackrc
configuration file for user account csback
sysop.csbackrc
configuration file for user account sysop
csbackresc.xml
configuration file for csbackresc.py
Notes as copied from BFO configuration directory
================================================
Hinweise zur csback-Konfiguration für das BFO
----------------------------------------------
Die Konfiguration wird ausführlich beschrieben unter:
http://anton2/trac/BFO/wiki/LAN/Rechner/pinatubo/csback
Die hier abgelegten *.csbackrc Dateien sind die Grundlage für die
crontab-Dateien. Die Konvertierung erfolgt mit dem Befehl csback2cron. Konkret
auszuführen ist:
csback2cron -l '$HOME/bin/' -v -o -i csback.csbackrc csback.crontab
csback2cron -l '$HOME/bin/' -v -o -i sysop.csbackrc sysop.crontab
Die so erzeugten crontab-Dateien werden auch im SVN abgelegt (als Referenz).
Zu beachten ist, dass bei der Installation auf prinatubo noch Passwörter in
die crontab-Dateien eingetragen werden müssen.
Dazu wird die crontab-Datei mit dem Befehl
crontab csback.crontab
geladen und das Passwort nach Aufruf von
crontab -e
eingetragen. Das Passwort soll auf keinen Fall in die Kopie eingetragen
werden, die im SVN liegt.
Außerdem muss eine BFO2LSDF-Zeile in csback.crontab manuell eingetragen
werden:
00 20 * * * $HOME/bin/BFO2LSDF >/dev/null 2>&1
----- END OF README -----
# This is <csbackrc>
# ----------------------------------------------------------------------------
# $Id: csback.csbackrc 1180 2015-05-11 08:20:41Z thof $
#
# Configuration file for csback for BFO backup archive.
#
# REVISIONS and CHANGES
# 26/04/2012 V1.0 Daniel Armbruster (seedlink configuration)
# 10/05/2012 V1.1 UIPC configuration (step 1 and 2)
# 15/09/2012 V1.2 Added HPMO testing rules.
# 06/02/2013 V1.3 Do not preserve timestamps.
# 07/03/2013 V1.4 use csbackscp when required
# 15/03/2013 V1.5 adjust exclude patterns in secure copy sections
# 04/03/2013 V1.6 adjust UIPC copy configuration
# 06/05/2013 V1.7 added entries for DL1 data and log
# 26/03/2014 V1.8 copy, backup and test for DL1 data from flocke
# 07/04/2014 V1.9 - only check subdirectories of
# /media/BFO_archive/DL1/data/thiesdl1
# - send notification messages to Peter too
# 11/05/2015 V1.10 exclude DL1memory from files being treated
#
# ============================================================================
#
# The appropriate command to generate a crontab file on stina to be installed
# on pinatubo later is:
#
# thof@stina:~/svnwork/BFO/docu/csback> ~/work/src/python/csback/csback2cron.py -o -l '$HOME/bin' -i csback.csbackrc csback.crontab
#
# Notice:
# The last line in the crontab file must be added manually:
#
# 00 20 * * * $HOME/bin/BFO2LSDF >/dev/null 2>&1
#
# thof, 26.3.2014
#
# ============================================================================
#
# Command to check the overall time schedule:
#
# thof@stina:~/svnwork/BFO/docu/csback> egrep '^\[|^cronexpr' csback.csbackrc
#
# ----------------------------------------------------------------------------
#
# Command to check keys against operational sections:
#
# thof@stina:~/svnwork/BFO/docu/csback> egrep '^\[|^keys' csback.csbackrc
#
# ============================================================================
#
# The file is organized as follows:
# - the keys for operational sections are defined first in the major section
# - subsequently the operational section are defined in the order of data
# streams: seedlink, UIPC, DL1, HPMO
#
# Recommendations for schedule
# ----------------------------
# Checksum integrity is verified once per month. Verification operations
# should start at 23:14 UT, each data source at a different day.
#
# Checksum creation takes place at:
# - seedlink ULP: 6:43 UT (configured in sysop.csbackrc)
# - seedlink: 4:02 UT (configured in sysop.csbackrc)
# - UIPC: 2:27 UT
# - DL1: 2:02 UT
#
# Copy operations should take place in the early morning, no interfering with
# checksum calculation.
#
# ============================================================================
#
# [copies]
#
# These sections control copy operations from primary data sources to
# /data/BFO as well as from /data/BFO to the USB-disk at /media/BFO_archive
#
# exceptions: - seedlink data copy to /data/BFO is controlled by seedlink
# - data copy to LSDF is controlled by shell script
# $HOME/bin/BFO2LSDF; the respective crontab entry must be added
# manually (see above)
#
[copies]
keys = seedlink, uipc1, uipc2, dl1, dl1log, dl1loglocal, dl1_2, dl1log_USB
#
# ----------------------------------------------------------------------------
#
# [backups]
#
# These sections control the calculation checksums and update of checksum
# files. Checksums are only created once at /data/BFO and then copyied along
# with the data to secondary storage media (USB disk and LSDF).
#
# exception: - checksums for seedlink data are calculated by a cron job in the
# user-space of user sysop
[backups]
keys = uipc1, DL1
#
# ----------------------------------------------------------------------------
# [tests]
#
# These sections control the verification of checksums in the local storage
# area at /data/BFO as well as on USB disk.
#
[tests]
keys = seedlink, uipc1, uipc2, hpmo1, hpmo2, DL1_1, DL1_2
# ----------------------------------------------------------------------------
# [mail]
#
# This section has no operational sub-section. It controls the dissemination
# of status messages by email.
#
# A password is required to send email trhough the T-Com SMTP server. This
# password must be added in the final crontab entry (not in the files commited
# to the subversion repository).
#
[mail]
cronexpr = @daily
sender = webmaster@bfo-obs.de
receivers = thomas.forbriger@kit.edu, widmer@geophys.uni-stuttgart.de, dani.armbruster@gmx.de, peter.duffner@kit.edu
host = smtp.webpage.t-com.de
port = 25
username = Webmaster@TB0001088541.webpage.t-com.de
password = XXXXXXXXXX
logging = no
days = 1
status = /var/log/csback.status
#
# ============================================================================
#
# Data: seedlink
# ==============
#
# IMPORTANT NOTE:
# Cause seedlink and slarchive are not providing the opportunity to dynamically
# change the group permissions for automatic generated archives, csbackgen must
# be executed by user sysop@pinatubo such that checksum files can be written
# to the corresponding directories in /data/BFO/seedlink/.
# Note: Copying and checking operations are perfomed as usual by user csback.
#
# copy local seedlink data from /data/BFO to USB storage
# ------------------------------------------------------
# some data is not to be included in the long-term archive and excluded from
# the copy operation through path pattern; the actual copy operation is done
# by the Unix cp command after csbackscp has checked the integrity of the
# source data
[copy_seedlink]
cronexpr = 17 3 * * *
srcdir = /data/BFO/seedlink/
targetdir = /media/BFO_archive/seedlink/
exclude = **/XM/BFO/H??**, **/XM/BFO/?L?**, **/XM/BFO/VM?**, **/XE/BFO/H??**, **/XE/BFO/BA1**, **/XX/, **/XS/BFO/?X6**, **/XS/BFO/[HB][GD]?**, **/XS/BFO/VM?**, **/XG/BFO/?X?**, **/XG/BFO/HG?**, **/XW/, .lock
specialcommands = cp --preserve=timestamps
secure = yes
#
# check integrity of seedlink data on USB storage
# -----------------------------------------------
[test_seedlink]
cronexpr = 14 23 1 * *
dir = /media/BFO_archive/seedlink/
tolerant = yes
logging = yes
recursive = yes
#
# ----------------------------------------------------------------------------
#
# Data: UIPC
# ==========
#
# copy UIPC data from graviton to local storage at /data/BFO
# ----------------------------------------------------------
[copy_uipc1]
cronexpr = 4 * * * *
srcdir = csback@graviton:/home/GWR_o056_local_copy/
targetdir = /data/BFO/SG/UIPC/
specialcommands = --size-only -rlpgoD
exclude = .lock, StripChartLog/, WORK/, WiFi_log/, POTENT.TSD, *.cfg, 2009o056/bin/, */CLC_o056/, */DAY_o056/, */FLT_o056/, */FTP_EMAIL_o056/, */MON_o056/, 2009o056/pub/, 2009o056/usr/, 2009o056/Upload/
keychain = yes
#
# copy UIPC data from local storage to USB storage
# ------------------------------------------------
# actual copy operation is done by Unix command cp; csbackscp only copies
# files with intact checksum integrity
[copy_uipc2]
cronexpr = 17 6 * * *
srcdir = /data/BFO/SG/UIPC/
targetdir = /media/BFO_archive/SG/UIPC/
specialcommands = cp --preserve=timestamps
exclude = .lock
secure = yes
#
# generate checksumfiles for local UIPC data at /data/BFO
# -------------------------------------------------------
# checksums are created from original data (mounted from UIPC Fit2PC using
# cifs)
[backup_uipc1]
cronexpr = 27 2 * * *
srcdir = /data/UIPC/
targetdir = /data/BFO/SG/UIPC/
exclude = .*StripChartLog.*, .*WORK.*, .*WiFi_log.*, .*POTENT.TSD, .*.cfg, .*2009o056.bin.*, .*CLC_o056.*, .*DAY_o056.*, .*FLT_o056.*, .*FTP_EMAIL_o056.*, .*MON_o056.*, .*2009o056.pub.*, .*2009o056.usr.*, .*2009o056.Upload.*
daystart= no
mtime = -1
hash = sha512
logging = yes
copy = no
test = no
#
# verify integrity of UIPC data in local storage at /data/BFO
# -----------------------------------------------------------
[test_uipc1]
cronexpr = 14 23 2 * *
dir = /data/BFO/SG/UIPC/
tolerant = yes
logging = yes
recursive = yes
#
# verify integrity of UIPC data in USB storage
# --------------------------------------------
[test_uipc2]
cronexpr = 14 23 3 * *
dir = /media/BFO_archive/SG/UIPC/
tolerant = yes
logging = yes
recursive = yes
#
# ----------------------------------------------------------------------------
#
# Data: DL1
# =========
#
# copy Thies DL1 data from recording host to local storage
# --------------------------------------------------------
[copy_dl1]
cronexpr = 23,53 * * * *
srcdir = dl1@flocke:/home/dl1/DL1/
targetdir = /data/BFO/DL1/
specialcommands = -a -rlpgoD
keychain = yes
#
# copy Thies DL1 system log files from recording host
# ---------------------------------------------------
[copy_dl1log]
cronexpr = 12 1 * * *
srcdir = dl1@flocke:/var/log/dl1*
targetdir = /data/BFO/DL1/log/flocke
specialcommands = -a --size-only
keychain = yes
#
# copy local Thies DL1 sytem log files
# ------------------------------------
[copy_dl1loglocal]
cronexpr = 22 1 * * *
srcdir = /var/log/dl1*
targetdir = /data/BFO/DL1/log/local
specialcommands = -a --size-only
keychain = no
#
# copy Thies DL1 data from local storage to USB storage
# -----------------------------------------------------
# this command only copies data with intact checksum integrity
# only data in thiesdl1 format is copied, ascii data is not archive
[copy_dl1_2]
cronexpr = 31 5 * * *
srcdir = /data/BFO/DL1/data/thiesdl1/
targetdir = /media/BFO_archive/DL1/data/thiesdl1/
specialcommands = cp --preserve=timestamps
exclude = .lock
secure = yes
#
# copy local Thies DL1 log files to USB storage
# ---------------------------------------------
# log files are not checksum verified
[copy_dl1log_USB]
cronexpr = 32 1 * * *
srcdir = /data/BFO/DL1/log/
targetdir = /media/BFO_archive/DL1/log/
specialcommands = -a --size-only
keychain = no
#
# generate checksum files for Thies DL1 data in local storage
# -----------------------------------------------------------
[backup_DL1]
cronexpr = 2 2 * * *
srcdir = /data/BFO/DL1/data/
targetdir = /data/BFO/DL1/data/
daystart= no
mtime = -1
hash = sha512
recursive = yes
logging = yes
# exclude current data and book-keeping file
exclude = .*DL1current.*, DL1memory
copy = no
test = no
#
# verify integrity of Thies DL1 data in local storage
# ---------------------------------------------------
[test_DL1_1]
cronexpr = 14 23 4 * *
dir = /data/BFO/DL1/data/
logging = yes
recursive = yes
#
# verify integrity of Thies DL1 data in USB storage
# -------------------------------------------------
[test_DL1_2]
cronexpr = 14 23 5 * *
dir = /media/BFO_archive/DL1/data/thiesdl1
logging = yes
recursive = yes
#
# ----------------------------------------------------------------------------
#
# Data: HPMO
# ==========
#
# HPMO data is static, since no new data is added to the collection.
# These sections control the verification of the integrity of archived data.
#
# verify integrity of hpmo data in /data/BFO
# ------------------------------------------
[test_hpmo1]
cronexpr = 14 23 6 * *
dir = /data/BFO/HPMO/
logging = yes
recursive = yes
#
# verify integrity of hpmo data on USB backup devices
# ---------------------------------------------------
[test_hpmo2]
cronexpr = 14 23 7 * *
dir = /media/BFO_archive/HPMO/
logging = yes
recursive = yes
#
# ------ END OF csback.csbackrc ------
<root>
<!-- global definitions -->
<general>
<user>sysop</user>
<seedlinkSrc>/home/sysop/seiscomp/acquisition/archive</seedlinkSrc>
<seedlinkTarget>/data/BFO/seedlink</seedlinkTarget>
</general>
<!-- start with seedlink stream definitions -->
<stream ID="XS">
<!-- <host>seis</host> -->
<host>192.168.3.71</host>
</stream>
<stream ID="XG">
<!-- <host>gamma</host> -->
<host>192.168.3.59</host>
</stream>
<stream ID="XM">
<!-- <host>magneton</host> -->
<host>192.168.1.64</host>
</stream>
<stream ID="XE">
<!-- <host>epsilon</host> -->
<host>192.168.3.73</host>
</stream>
<stream ID="XW">
<!-- <host>floh</host> -->
<host>192.168.3.38</host>
</stream>
<stream ID="GR">
<!-- <host>bfo-grsn2</host> -->
<host>192.168.3.37</host>
</stream>
</root>
# This is <sysop.csbackrc>
# ----------------------------------------------------------------------------
# $Id: sysop.csbackrc 1022 2013-11-09 13:01:20Z damb $
#
# Configuration file for csback for BFO backup archive.
#
# REVISIONS and CHANGES
# 07/05/2012 V1.0 Daniel Armbruster (seedlink checksumfile generation
# configuration)
# 21/05/2012 V1.1 U-data (1/100 Hz) will be handled separately
#
# ============================================================================
#
# IMPORTANT NOTE:
# * Cause seedlink and slarchive are not providing the opportunity to
# dynamically change the group permissions for automatic generated archives,
# csbackgen must be executed by user sysop so that checksumfiles can be written
# into the corresponding directories in /data/BFO/seedlink/.
# * Checking the integrity and copying the data is performed by user csback on
# pinatubo.
# * U (1/100 Hz) data will be handled separately cause the file could be older
# than 24 hours but because mseed files were written blockwise it might occur
# that a seedlink-record hadn't been completed yet within these 24 hours.
# csbackgen would compute a checksum for this datafile even though the file
# will be modified afterwards.
[backups]
keys = seedlink, seedlink_udata
# this entry is responsible for generating the checksumfiles for all seedlink
# stream files except for U-data streams (0.01 Hz)
[backup_seedlink]
cronexpr = 2 4 * * *
srcdir = /data/BFO/seedlink/
targetdir = /data/BFO/seedlink/
daystart= no
mtime = -1
hash = sha512
recursive = yes
logging = yes
# exclude U-data
exclude = .*\.U\w\w\.D\.\d\d\d\d\.[0-3][0-9][0-9]$
copy = no
test = no
# U-data seedlink streams are handled separately (see also comment above)
[backup_seedlink_udata]
cronexpr = 43 6 * * *
srcdir = /data/BFO/seedlink/
targetdir = /data/BFO/seedlink/
daystart= no
mtime = -2
hash = sha512
recursive = yes
logging = yes
copy = no
test = no
# ------ END OF sysop.csbackrc ------
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment