Commit eed9d3f6 authored by Daniel Armbruster's avatar Daniel Armbruster Committed by thomas.forbriger
Browse files

documentation complete; corrections

This is a legacy commit from before 2015-05-18.
It may be incomplete as well as inconsistent.
See COPYING.legacy and README.history for details.


SVN Path:     http://gpitrsvn.gpi.uni-karlsruhe.de/repos/TFSoftware/trunk
SVN Revision: 4408
SVN UUID:     67feda4a-a26e-11df-9d6e-31afc202ad0c
parent 317d6342
......@@ -70,7 +70,7 @@ class Usage(Error):
"\n"+__subversion__+"\nAuthor: "+__author__+ """
Usage: csbackgen [-v|--verbose] [-e REGEX [-e REGEX [...]]]
[-R|--notrecursive] [-d|--debug] [-f|--followlinks]
[-t|--target DIR] [-l|--logging] [--hash ARG]
[-t|--target DIR] [-l|--logging] [-H|--hash ARG]
[-L|--lock] PATH
or: csbackgen -h|--help\n"""
sys.stderr.write("csbackgen: " + self.msg + "\n")
......@@ -85,7 +85,7 @@ def help():
__subversion__+"\nAuthor: "+__author__+"""
Usage: csbackgen [-v|--verbose] [-e REGEX [-e REGEX [...]]]
[-R|--notrecursive] [-d|--debug] [-f|--followlinks]
[-t|--target ROOTDIR] [-l|--logging] [--hash ARG]
[-t|--target ROOTDIR] [-l|--logging] [-H|--hash ARG]
[-L|--lock] PATH
or: csbackgen -h|--help
-------------------------------------------------------------------------------
......@@ -104,7 +104,7 @@ def help():
have the same subdirectory structure as PATH.
-l|--logging Switch on logging to files. Logfiles will be located in
/var/log/ .
--hash ARG Set the hash function algorithm. Valid values are:
-H|--hash ARG Set the hash function algorithm. Valid values are:
sha224, sha256, sha384, sha512. (default: sha256)
-L|--lock Lock the directories working at. This flag is useful in
case csbackgen was run simultaneously with other csback
......@@ -140,9 +140,9 @@ def main(argv=None):
argv = sys.argv
try:
try:
opts, args = getopt.getopt(argv[1:], "vhe:Rdft:lL", ["help", "verbose", \
"notrecursive", "debug", "followlinks", "target=", "logging", \
"hash=", "lock"])
opts, args = getopt.getopt(argv[1:], "vhe:Rdft:lLH:", ["help",
"verbose", "notrecursive", "debug", "followlinks", "target=", \
"logging", "hash=", "lock"])
except getopt.GetoptError as err:
raise Usage(err.msg)
verbose = False
......@@ -174,7 +174,7 @@ def main(argv=None):
targetDirectory = arg.rstrip(os.sep)+os.sep
elif opt in ("-l", "--logging"):
logger.configure()
elif opt in ("--hash",):
elif opt in ("-H", "--hash"):
if arg in VALIDHASHES:
hashfunc = arg
else:
......
......@@ -13,7 +13,8 @@
# REVISIONS and CHANGES
# 11/09/2011 V1.0 Daniel Armbruster
# 08/01/2012 V1.1 new configuration file syntax
# 11/01/2012 v1.2 copy sections introduced
# 11/01/2012 V1.2 copy sections introduced
# 16/01/2012 V1.3 hash entry for backup section rule
#
#
# ============================================================================
......@@ -84,6 +85,7 @@
# srcdir = /data3/
# exclude = .*.log$, .*tmp[1-5], ^dirname$
# targetdir = /data3.bak/
# hash = sha512
# recursive = yes
# followlinks = yes
# test = yes
......@@ -116,8 +118,8 @@
# port SMTP port.
# username Username for SMTP server login.
# password Password for SMTP server login.
# logging Switch on the logging mechanism to the csback logfiles in
# ~/.csback/log/ . (default no)
# logging Switch on the logging mechanism to the csback logfile in
# /var/log/ . (default no)
#
#
# Copies, Backups and Tests:
......@@ -155,6 +157,8 @@
# targetdir Directory path to backup data.
# exclude Comma (followed by a whitespace) seperated regular expressions
# of files/directories to be excluded from the backup.
# hash Set the hash function algorithm to one of the following values:
# sha224, sha256, sha384, sha512. (default sha256)
# recursive Include files of subdirectories. (default yes) Refers as well
# to the copy command if copy enabled.
# logging Switch on the logging mechanism to the csback logfile in
......
......@@ -70,7 +70,7 @@
Daniel Armbruster\\
Blackforest Observatory (BFO)\\
\vspace{3pt}
\today\ (\mbox{\texttt{$$Revision: 0.1 $$}})\par
\today\ (\mbox{\texttt{$$Revision: 0.2 $$}})\par
\end{raggedleft}
\bigskip]
\thispagestyle{plain}
......@@ -83,9 +83,9 @@ Blackforest Observatory (BFO)\\
The \texttt{csback} utility is a collection of small python scripts which
provide the possibility to calculate \emph{checksum files} and check the
checksums of the corresponding files from time to time. Because all information
are saved in \texttt{ASCII} files the processes always are transparent. This
is saved in \texttt{ASCII} files the processes always are transparent. This
manual describes the installation and configuration of \texttt{csback} which was
developed by Daniel Armbruster.
developed by \href{mailto:dani.armbruster@gmail.com}{Daniel Armbruster} .
\texttt{csback} was developed both for python2 and python3 and therefore should
be platform independent. Because it uses several already existing unix tools
......@@ -125,7 +125,7 @@ checksum file.
\item \texttt{csbackchk}: Check the checksums of files listed in the checksum
file. Notice that \texttt{csbackchk} does not check if there are new files in
the directory existing.
\item \texttt{csbackntfy}: Check the \texttt{csback} logfiles and send a email
\item \texttt{csbackntfy}: Check the \texttt{csback} logfile and send a email
to the admin that contains the current status.
\end{itemize}
\section{Installation}
......@@ -133,11 +133,9 @@ to the admin that contains the current status.
To install the \texttt{csback} toolkit just use the provided \emph{Makefile} and
enter
\medskip
\texttt{make install}
\medskip
\begin{verbatim}
make install
\end{verbatim}
to the command line. If you prefer the installation by hand just guarantee that
your environment knows the \texttt{csback} scripts. First make the scripts
executable. Then use the appropriated names for the softlinks your are creating
......@@ -151,22 +149,21 @@ Before using \texttt{csback2cron} to generate a \emph{crontab} a
contains such a configuration file that contains a description of available
commands and configuration options. Afterwards either \texttt{csback2cron} has
to be called with the command
\medskip
\texttt{csback2cron -i path/to/configfile crontab}
\medskip
\begin{verbatim}
csback2cron -i path/to/configfile crontab
\end{verbatim}
or before executing \texttt{csback2cron} a \texttt{csback} configuration file
must be copied to \texttt{\$HOME/.csback/csbackrc} so that the command
\medskip
\texttt{csback2cron crontab}
\medskip
\begin{verbatim}
csback2cron crontab
\end{verbatim}
is sufficient to generate a \emph{crontab} file with the filename
\texttt{crontab}. Then install the \texttt{crontab} using the appropriated
\emph{crontab} command. Note that a exemplary \texttt{csback} configuration file
comes along with this package.
\emph{crontab} command. Note that an exemplary \texttt{csback} configuration
file comes along with this package.
While converting the \texttt{csback} configuration file entries to
\texttt{crontab} lines \texttt{csback2cron} does not perform any logical checks.
\subsection{Configuration file syntax}
A \texttt{csback} configuration file may include comments, prefixed by specific
......@@ -182,14 +179,12 @@ accepted. Note that leading whitespace is removed from values.
\subsection{Configuration parameters}
\subsubsection{\texttt{copy} configuration}
Within a \texttt{[copy]} section a copy rule section headers can be defined. The
Within a \texttt{[copy]} section copy rule section headers can be defined. The
only option which is available within a \texttt{[copy]} section is the
\texttt{keys} entry. An exemplary line looks like
\medskip
\texttt{keys = copy1, copy2, copy3}
\medskip
\begin{verbatim}
keys = copy1, copy2, copy3
\end{verbatim}
which defines three copy rule sections named \texttt{copy1}, \texttt{copy2} and
\texttt{copy3}. Now the three copy rule sections must be defined.
\begin{verbatim}
......@@ -215,9 +210,9 @@ Every copy rule section's name includes the prefix \texttt{copy\_}. A minimal
copy rule section setup is \texttt{[copy\_copy3]}. It only contains obligatory
entries that are:
\begin{itemize}
\item \texttt{cronexpr}: Here the copy process can be scheduled. A valid cron
expression should be provided. A good introduction to cron expressions is
\url{http://en.wikipedia.org/wiki/Cron}.
\item \label{item:cronexpr} \texttt{cronexpr}: Here the copy process can be
scheduled. A valid cron expression should be provided. A good introduction to
cron expressions is \url{http://en.wikipedia.org/wiki/Cron}.
\item \texttt{srcdir}: Directory path to source files.
\item \texttt{targetpath}: Directory path for copied source files.
\end{itemize}
......@@ -227,8 +222,8 @@ entries are:
\item \texttt{exclude}: A comma separated list of patterns for files and
directories which were excluded if matching one of the patterns. Because the
\texttt{csback} toolkit falls back to \texttt{rsync} to copy files read the
\texttt{rsync} man page \url{http://rsync.samba.org/ftp/rsync/rsync.html} for
more information on patterns.
\texttt{rsync} \href{man page}{http://rsync.samba.org/ftp/rsync/rsync.html} for
more information on available patterns. \label{item:copy_exclude}
\item \texttt{specialcommands}: A whitespace separate list of patterns for
additional \texttt{rsync} options.
\end{itemize}
......@@ -241,21 +236,18 @@ calculated of files located in \texttt{srcdir} and afterwards check the
checksums of files backed up in \texttt{targetdir} with the checksums available
in the checksum file generated before. Though both the copy process and the
check process might be switched off it is recommended to proceed in this order
to make sure that files backed up are registered in the corresponding checksum
file.
to make sure that files backed up were copied correctly and registered in the
corresponding checksum file properly.
As in a \texttt{[copy]} section in a \texttt{[backup]} section backup rule
section headers can be defined. The only entry which can be defined is the
\texttt{keys} attribute. A exemplary back \texttt{[backup]} section will look
as follows:
\texttt{keys} attribute. A exemplary \texttt{[backup]} section looks as
follows:
\begin{verbatim}
[backup]
keys= backup1, backup2, backup3
\end{verbatim}
Next the appropriated backup section rules must be set up.
\begin{verbatim}
[backup_backup1]
cronexpr = * */10 * * *
......@@ -286,32 +278,147 @@ targetdir = /data3.bak/
recursive = yes
followlinks = yes
test = yes
hash = sha512
tolerant = no
\end{verbatim}
As already within the \texttt{[copy]} rule sections within a \texttt{[backup]}
As already within the \texttt{[copy]} rule sections in a \texttt{[backup]}
rule section there are obligatory entries and auxiliary entries. Obligatory
entries are:
\begin{itemize}
\item \texttt{cronexpr}: Cron expression. See \dots
\item \texttt{cronexpr}: Cron expression. See section \ref{item:cronexpr}.
\item \texttt{srcdir}: Directory path to source files.
\item \texttt{targetdir}: Directory path for backup files.
\end{itemize}
Entries which already have a default value and so are customizable if desired
Entries which already have a default value and so are customizable optionally
are:
\begin{itemize}
\item \texttt{copy}: Setting the value to \texttt{yes} will enable a copy
process with \texttt{rsync} before generating or rather updating checksum files.
\item \texttt{copy-exclude}:
\item \texttt{recursive}:
\item \texttt{followlinks}:
\item \texttt{test}:
\item \texttt{tolerant}:
\end{itemize}<++>
Default value is \texttt{no}.
\item \texttt{copy-exclude}: Provides the same functionality as the
\texttt{exclude} key in a \texttt{[copy]} rule section in section
\ref{item:copy_exclude}.
\item \texttt{exclude}: A comma separated list of \emph{regular expressions}.
Files or directories matching one of those regular expressions will be excluded
both from the checksum file generation or rather updating process and the
integrity check afterwards. Notice that the syntax of the regular expressions
passed here differs from the patterns passed in the \texttt{copy-exclude} key.
If you are not familiar with regular expressions read
\url{http://en.wikipedia.org/wiki/Regular_expression} or
\url{http://www.regular-expressions.info/}. \label{item:exclude}
\item \texttt{recursive}: Recursive mode. If set to \texttt{yes} generating
checksum files for subdirectories is enabled. Notice that this key refers as
well to \texttt{rsync} if the \texttt{copy} key from above is set to
\texttt{yes}. Default value is \texttt{yes}. \label{item:recursive}
\item \texttt{followlinks}: If a directory contains links and this key is set to
\texttt{yes} then files will be listed to the checksum file. \texttt{csbackgen}
will generate checksum files for directories linked into \texttt{srcdir} or one
of its subdirectories. (If the \texttt{recursive} key is set to \texttt{yes}.)
Notice that this key refers as well to \texttt{rsync} if the \texttt{copy} key
from above is set to \texttt{yes}. Default value is \texttt{no}.
\item \texttt{hash}: Select a hash function algorithm for \texttt{csbackgen}.
Available values are \texttt{sha224}, \texttt{sha256}, \texttt{sha384} and
\texttt{sha512}. Default is \texttt{sha256}. \label{followlinks}
\item \texttt{test}: Enable check of the copied files integrity. Default value
is \texttt{yes}.
\item \texttt{tolerant}: Be tolerant. If set to \texttt{yes} \texttt{csbackchk}
only will issue a \emph{WARNING} to the system logger if a file listed in a
checksum file is not available anymore. Else a \emph{ERROR} will be send. This
key might be useful if \texttt{csback} is used on a filesystem that works as a
ring buffer. Default is \texttt{no}. Notice that the \texttt{csback} logging
mechanism must be enabled by setting the \texttt{logging} key from below to
\texttt{yes}. \label{item:tolerant}
\item \texttt{logging}: Enable the \texttt{csback} logging mechanism. Log
reports will be send to the system logger. A default \texttt{syslog-ng V3.0}
configuration file comes along with \texttt{csback} which send the log messages
to a file named \texttt{csback.log} located in \texttt{/var/log/}. Default value
is \texttt{yes}. Logging is strongly recommended. For an advanced
\texttt{syslog-ng} configuration read the
\href{http://www.balabit.com/dl/guides/syslog-ng-v3.0-guide-admin-en.pdf}
{\emph{syslog-ng Administrator Guide}}. \label{item:logging}
\end{itemize}
\subsubsection{\texttt{test} configuration}
\subsubsection{\texttt{mail} configuration}
\texttt{[test]} rule sections are provided to test the files integrity form time
to time. The checksums of files listed in its directory's checksum file will be
compared to the registered checksum and so either a \emph{WARNING}, a
\emph{ERROR} or a \emph{CRITICAL} log message will be send to the systems logger
if \texttt{logging} is enabled. First the test rule section headers must be
defined using the \texttt{keys} option within a \texttt{[test]} section. For
example:
\begin{verbatim}
[tests]
keys = test1, test2
\end{verbatim}
so that the list of test rule section headers is comma separated. Then the
exemplary test rule sections must be defined.
\begin{verbatim}
[test_test1]
cronexpr = 50 3 * * *
dir = /data3.bak/
tolerant = yes
[test_test2]
cronexpr = @daily
srcdir = /media/srcdir/
dir = /data2.bak/
tolerant = yes
recursive = no
logging = no
\end{verbatim}
Obligatory keys are:
\begin{itemize}
\item \texttt{cronexpr}: For a description see \ref{item:cronexpr}.
\item \texttt{dir}: The checksums of registered files will be compared to the
checksum available in the checksum file. If \texttt{logging} is enabled
\texttt{csback} will either send an \emph{ERROR} or a \emph{CRITICAL} message to
the system logger if the check was not successful.
\end{itemize}
Additionally there are some optional keys which include:
\begin{itemize}
\item \texttt{srcdir}: Optional source directory of files and subdirectories
(if the \texttt{recursive} key is set to \texttt{yes}) in which their checksums
will be compared to the checksum file located in \texttt{dir} or one of its
subdirectories. Notice to perform a test successfully the subdirectory structure
of \texttt{dir} and \texttt{srcdir} must be equal.
\item \texttt{exclude}: Exclude files or subdirectories of tests matching one of
the comma separated \emph{regular expressions} passed here. For further
information see section \ref{item:exclude}.
\item \texttt{recursive}: Disable testing integrity of files located in
subdirectories of \texttt{dir}. See also \ref{item:recursive}.
\item \texttt{followlinks}: While testing follow links both to subdirectories
and files. See also \ref{followlinks}.
\item \texttt{tolerant}: See \ref{item:tolerant}.
\item \texttt{logging}: Enable the \texttt{csback} logging mechanism. See also
\ref{item:logging}.
\end{itemize}
\subsubsection{\texttt{mail} configuration}
Within a \texttt{[mail]} section the \texttt{csback} logfile checking mechanism
can be configured. There are nearly only obligatory keys. So if a \texttt{[mail]}
section in a \texttt{csback} configuration file had been defined all keys must
be set carefully to guarantee a correct functionality. \texttt{csbackntfy} does
not work yet together with a local SMTP server.
\begin{itemize}
\item \texttt{cronexpr}: Cron expression to schedule the \texttt{csback} notify
mechanism. See also \ref{item:cronexpr}.
\item \texttt{sender}: Email address of the sender.
\item \texttt{receivers}: Comma separated list of receiver email addresses.
\item \texttt{host}: Hostname of the SMTP server.
\item \texttt{port}: Port of the SMTP server.
\item \texttt{username}: Username for SMTP server.
\item \texttt{password}: Password of \texttt{username} for SMTP server. Remember
to make the \texttt{csback} configuration file readonly to prevent abusing the
password.
\end{itemize}
The only optional key within a \texttt{[mail]} section is the \texttt{logging}
key. If logging for \texttt{csbackntfy} is desired set the value to
\texttt{yes} else to \texttt{no}. Default value is \texttt{no}.
The result of the checks performed will be send to a logfile named
\texttt{checksumfile.result} which will be created in the directory the checksum
file is located.
\newpage
\onecolumn
\begin{appendix}
\section{Programs}
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment