|
|
# Legacy documentation (copied from TFSoftware wiki)
|
|
|
(XXX macro: "PageOutline")
|
|
|
|
|
|
Navigation: [[Mainpage], [wiki:trunk], [wiki:branches], [wiki:tags], [wiki:vendor], [wiki:export], [wiki:admin], [wiki:docs|WikiStart]]
|
|
|
[Origin of this file](http://gpitrsvn.gpi.uni-karlsruhe.de:8000/TFSoftware/wiki/trunk/src/python/csback)
|
|
|
|
|
|
`csback` is a software suite written in **Python** http://www.python.org/ to manage backups and test their integrity from time to time.
|
|
|
|
|
|
## Open issues
|
|
|
[[TicketQuery(status!=closed&status=accepted&component=csback,order=id,desc=1,format=table,col=owner|summary|priority|reporter|component)]]
|
|
|
|
|
|
## Aims
|
|
|
## Design goals
|
|
|
* Use already existing tools and programs which are already well tested to avoid writing the same application just in another software language a second time.
|
|
|
* Follow the Unix philosophy and write for each task one tool.
|
|
|
* Software should work both under Python2 and Python3 to avoid compatibility problems. Successfully tested both under Python 2.7 and Python 3.x.
|
|
|
|
|
|
|
|
|
## Notes
|
|
|
Notes, ideas and thoughts due to the implementation of the `csback` software suite.
|
|
|
Notes, ideas and thoughts regarding the implementation of the `csback`
|
|
|
software suite.
|
|
|
* Overview:
|
|
|
1. `csback` uses a central node to generate or rather update checksumfiles. This node must have access (up to now via NFS) to all data which should be backed up. An advantage of this approach is to avoid great traffic and intensive calculation processes on low resource systems.
|
|
|
2. In an exemplary backup sequence `csback` applies the following process sequence:
|
... | ... | @@ -47,32 +43,32 @@ Notes, ideas and thoughts due to the implementation of the `csback` software sui |
|
|
Only perform a integrity check of files as defined within a `test` section.
|
|
|
d. `mail`:
|
|
|
Specify the mail settings for `csbackntfy.py`.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackgen.py]
|
|
|
* csbackgen.py:
|
|
|
1. Generate or rather update (if already existing) a checksumfile. Only append to checksumfiles so already existing lines never are touched again.
|
|
|
2. If a specified directory contains subdirectories each subdirectory will contain its own checksumfile.
|
|
|
3. Up to now hash functions of the SHA-2 family (SHA-224, SHA-256, SHA-384, SHA-512) are allowed.
|
|
|
4. Generating a checksumfile with arbitrary source files (located in a different source) is provided.\\
|
|
|
If logging is enabled log messages will be sent to the systemlogger of the operating system on tcp port 3333.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackchk.py]
|
|
|
* csbackchk.py:
|
|
|
Check the integrity of a file that is registered in a `csback` checksumfile.\\
|
|
|
This program will take either one or two arguments. In case of passing two arguments `csbackchk` will take the checksumfile of the directory passed as the second argument (`PATH`) and compare its checksums with the checksums of the files located in the directory passed as first argument (`SOURCEPATH`). In case only one argument had been passed `csbackchk.py` will perform a integrity check of the files located in `PATH`. (In both cases a checksumfile **must** be located in `PATH`.)\\
|
|
|
If logging is enabled log messages will be sent to the systemlogger of the operating system on tcp port 3333.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackntfy.py]\\
|
|
|
* csbackntfy.py:
|
|
|
Perform checks of the `csback` logfile and report the current status to the admin via email or using its tiny nagios interface.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackobs.py]\\
|
|
|
Small observer script which reports the exit status of external programs to the `csback` logfile in case of failure. Up to now only in use for `rsync`.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csfile.py]\\
|
|
|
* csfile.py:
|
|
|
Module which provides a class to read, write and treat with a checksum file.
|
|
|
Tools which will make use of this module are:
|
|
|
1. `csbackgen.py`
|
|
|
2. `csbackchk.py`
|
|
|
Additionally this module contains the functionality to exclude files regarding there access/change or modification times.
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbacklog.py]\\
|
|
|
* csbacklog.py:
|
|
|
Module that handles the `csback` logging mechanisms. The module provides as well a `[[ResultLogger]]` class which is in use of `csbackchk.py` to generate `checksumfile.result` files.
|
|
|
|
|
|
## Checksum file format
|
|
|
|| checksum || path || sha-2 algorithm || creation date file || creation location checksum || creation date of checksum ||
|
|
|
|| 65135b861519486f3fc171b8efcf7078d123b0d8b3cdcb305227facf8eae7659 || data/datafile || sha256 || 2007/15/09-00:22:10 || pinatubo || 2007/19/09-00:21:17 ||
|
|
|
| checksum | path | sha-2 algorithm | creation date file | creation location checksum | creation date of checksum |
|
|
|
| 65135b861519486f3fc171b8efcf7078d123b0d8b3cdcb305227facf8eae7659 | data/datafile | sha256 | 2007/15/09-00:22:10 | pinatubo | 2007/19/09-00:21:17 |
|
|
|
|
|
|
With that a typical checksum file line will look as follows:
|
|
|
|
... | ... | @@ -90,11 +86,11 @@ See as well ticket:158. The handling of saving the checking results changed. |
|
|
|
|
|
|
|
|
## Todo
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csback2cron.py]
|
|
|
* csback2cron.py:
|
|
|
1. ~~Implement a private function to test the cron timestamp in the Processor class.~~ No checks of the timestamps are performed. This task is delegated to the `cron deamon`.
|
|
|
2. ~~Implement the possibility to change the frequency which uses `csbackntfy` to report the current status.~~ Scheduling now works with a usual cron timestamp.
|
|
|
3. ~~Allow the user to check the files of a directory with an arbitrary source directory.~~
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackgen.py]
|
|
|
* csbackgen.py:
|
|
|
1. ~~Implement the possibility to generate a checksumfile in a target with files of any source directory.~~
|
|
|
2. ~~Use here the logging module of python to report the current exit status. A great feature. It's convenient for debugging and verbosity, too.~~
|
|
|
3. ~~Use the PID lockfilehandler.~~ See ticket:153
|
... | ... | @@ -102,7 +98,7 @@ See as well ticket:158. The handling of saving the checking results changed. |
|
|
5. ~~File exclusion equal to unix `find` command.~~
|
|
|
6. Implement ssh-client feature. See ticket:154
|
|
|
7. Make use of *par2* hash algorithm. See ticket:165
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackchk.py]
|
|
|
* csbackchk.py:
|
|
|
1. ~~Use here the logging module of python to report the current exit status.~~
|
|
|
Exit status will be reported in so called `checksumfile.result` files. A exemplary line will look as follows:
|
|
|
|
... | ... | @@ -111,7 +107,7 @@ See as well ticket:158. The handling of saving the checking results changed. |
|
|
|
|
|
2. ~~Use the PID lockfilehandler.~~ See ticket:153
|
|
|
3. Implement ssh-client feature. See ticket:154
|
|
|
* [source:tfsoftware/trunk/src/python/csback/csbackntfy.py]
|
|
|
* csbackntfy.py:
|
|
|
1. ~~Parse csback logfile and report the status either via mail or using the nagios interface.~~
|
|
|
2. It would be nice to have options for a local SMTP server.
|
|
|
|
... | ... | @@ -125,7 +121,7 @@ Python specific: |
|
|
* Regular expression module: http://docs.python.org/library/re.html
|
|
|
* Basic tutorial for logging module: http://docs.python.org/howto/logging.html#logging-basic-tutorial
|
|
|
* Advanced tutorial for logging module: http://docs.python.org/howto/logging.html#logging-advanced-tutorial
|
|
|
(XXX macro: "Top")
|
|
|
|
|
|
## Hints
|
|
|
### Bookkeeping
|
|
|
Possibly csback will require some bookkeeping facility.
|
... | ... | @@ -135,4 +131,4 @@ has a kind of bookkeeping device in the class Memory. |
|
|
The source code can be found in [[and [source:trunk/src/conv/ThiesDL1/memory.cc|source:trunk/src/conv/ThiesDL1/memory.h]]].
|
|
|
It is used in [source:trunk/src/conv/ThiesDL1/DL1logger.cc@3084:349-370#L346].
|
|
|
|
|
|
(XXX macro: "Top") |
|
|
|