Commit e94f5def authored by thomas.forbriger's avatar thomas.forbriger Committed by thomas.forbriger
Browse files

merged trunk changes

This is a legacy commit from before 2015-03-01.
It may be incomplete as well as inconsistent.
See COPYING.legacy and README.history for details.

SVN Path:
SVN Revision: 3483
SVN UUID:     67feda4a-a26e-11df-9d6e-31afc202ad0c
parent b45140ce
# this is <DL1_logrotate.conf>
# ============================================================================
# config file for logrotate
# -------------------------
# $Id$
# ============================================================================
# This file should be installed in /etc/logrotate.d on OpenSuSE systems
# ============================================================================
# used by all syslog daemons
/var/log/dl1log /var/log/dl1logall {
maxage 365
size +4096k
create 644 root root
/etc/init.d/syslog reload
# ----- END OF DL1_logrotate.conf -----
% this is <DL1recording.tex>
% ----------------------------------------------------------------------------
% $Id: DL1recording.tex,v 1.12 2010/10/15 14:44:22 tforb Exp $
% Copyright (c) 2008 by Thomas Forbriger (BFO Schiltach)
% Recording data from the DL1 data logger
% 22/12/2008 V1.0 Thomas Forbriger
% 15/10/2010 V1.1 comment on message() versus match() filter in
% syslog-ng
% ============================================================================
\newcommand{\mytitle}{Recording data from the Thies DL1 pluviometer}
\rfoot[Black Forest Observatory, \today]{Thomas Forbriger}
\lfoot[Thomas Forbriger]{Black Forest Observatory, \today}
\parskip 10pt
% macros
\newcommand{\DTDL}{Thies DL1 data logger}
\mbox{ }\par
Technical Documentation
Thomas Forbriger\\
Black Forest Observatory (BFO)\\
Heubach 206,
D-77709 Wolfach\\
\today\ (\mbox{\texttt{$$Revision: 1.12 $$}})\par
\parskip 0pt
\parskip 10pt
\section{Pluviometer data acquisition}
\subsection{Hardware components}
The rain gauge (a tiping bucket) in front of the BFO laboratory building
produces an eletrical pulse for each 0.1\,mm of precipitation.
These pulses are counted by a \DTDL.
The \DTDL\ maintains a data ring-buffer.
Its contents can be accessed through the local display, a memory card, or the
RS-232 serial port.
We use a Linux machine (\Dloggerhost) connected to the serial port to control
the \DTDL\ and to collect precipitation data.
\subsection{Data acquisition program}
Two programs are currently available to control the \DTDL.
They are \DLdirect\ (Section~\ref{sec:prog:dldirect}) and \DLlogger\
The first supports manual control of the \DTDL\ and should not be used while
the second is in operation.
The second (\DLlogger) is the actual data acquisition program.
It controls the \DTDL, synchronizes its internal clock to the system clock of
the logger host (\Dloggerhost), and maintains an archive of data.
See Section~\ref{sec:prog:dllogger} for a detailed description in particular
for information on data sampling, poll cycles, message logging, and so on.
The data acquisition program (\DLlogger) is started from a shell script
(\Dlogscript, see Section~\ref{sec:prog:dlogscript}).
This shell script sets the correct values for the data directories, creates
them, and initializes the serial port prior to execution of \DLlogger.
The script \Dlogscript\ itself is executed from the launch-script
\Dlaunchscript\ (see Section~\ref{sec:prog:dlaunchscript}).
The latter first checks whether an instance of the logger is in operation.
If not, the logger is executed in background operation mode.
The launch-script \Dlaunchscript\ should be executed periodically by the cron
See Section~\ref{sec:conf:crontab:logger} for an example of an entry in the
This ensures that the data acquisition is resumed automatically upon
unexpected termination of the acquisition program.
The data acquisition program writes its diagnostic messages to the system's
syslog facility.
The messages are tagged \DLlogger.
See Section~\ref{sec:prog:dllogger} for information on log levels used for
different diagnostic messages.
A simple configuration that maintains local publically readable log files
is given in Section~\ref{sec:conf:syslog:logger}.
This configuration also sends messages to a second system for monitoring
Two files are maintained.
One contains all messages that are tagged \DLlogger.
These messages comprise alos purely informational messages that are issued in
each poll cycle to indicate the activity of \DLlogger.
This log stream is also exported to the display host \Ddisplayhost\ for
monitoring purposes.
A second log file contains only messages with level notice and higher (all
levels except level info).
This log file is meant to be archived together with the data to support error
Log messages with tag \DLlogger\ are excluded from the usual system log.
\subsection{Data archive}
The data archive maintained by \DLlogger\ contains two types of data files.
The first are called dump-files and are simply literal copies of the data
received from the \DTDL\ together with some diagnostics.
The second kind of data files uses GSE format, contains diagnostics too and
is ready to be processed by plotting facilities.
The GSE data contains data in counts, not mm.
Both data archives contain files for completed days that provide the date in
the file name (as well as in the file's header).
The current data for uncompleted days is stored in files called
\texttt{active.asc} and \texttt{active.gse} for the dump and the GSE data
The latter are updated in periodic poll intervals.
The data archive can be provided to other hosts via network file system (NFS).
An example export entry is provided in Section~\ref{sec:conf:exports:logger}.
The sampling interval provided in the GSE data files is not exact, since GSE
specifies a data rate rather than an interval.
See Section~\ref{sec:prog:dllogger} for further comments.
Once a day the data logger \DLlogger\ synchronizes the \DTDL\ to the system
clock of \Dloggerhost.
See Section~\ref{sec:prog:dllogger} for further comments on timing accuracy.
The system clock of \Dloggerhost\ must by disciplined by NTP (network time
To support easy monitoring of the systems state of health a shell script
\Dntpscript\ (Section~\ref{sec:prog:dlntpscript}) is executed once a day by
the cron deamon (see Section~\ref{sec:conf:crontab:logger}).
This script polls the state of NTP peers and writes it to the system log
stream with tag \DLlogger.
The system's time zone is selected to be UTC.
This way the time stamps in the system's log files are synchronous with the
times provided in the messages from \DLlogger.
\subsection{Data acquisition account}
The logger program \DLlogger\ runs in userspace of user \Dloggeruser\ on
The home directory of \Dloggeruser\ is \Dloggeruserhome.
Binaries \DLlogger\ and \DLdirect\ are installed in
Shell scripts \Dlogscript, \Dlaunchscript, and \Dntpscript\ are installed in
The \DLlogger\ memory file, the data files, and logs of the launch script go
to \Dloggeruserhome\texttt{/data/DL1} and subdirectories therein.
\subsection{Data stream}
\DLlogger\ uses the following identifiers for the data steam in GSE files:
\item[channel:] \Dchannel
\item[station:] is read from \DTDL
\item[instype:] is read from \DTDL
The channel ID is defined according to the SEED reference manual (Appendix~A):
\item[Band code W:] Weather/Environmental
\item[Instrument Code R:] Rainfall
\item[Orientation Code:] unkown
\section{Data monitoring}
A second host (\Ddisplayhost) may be used for the purpose of monitoring, data
quality control, and data backup.
Data are made available through an NFS mount
Log messages are made available through syslog in a local file
\subsection{Data graph monitor}
The data files are made available through NFS.
The processing and plotting mechanism is described elsewhere in a difference
\subsection{Data file monitor}
For calibration purposes in particular a direct display of the current data
file can be useful.
The file can be displayed in a local terminal window by using the command
\texttt{less /data/DL1/dump/active.asc}
The display will be updated upon pressing the \texttt{R}-key.
\subsection{Message file monitor}
A permanent display of diagnostic messages can be maintained in a local
terminal window by the command
\texttt{less /var/log/dl1log}
Pressing the \texttt{F}-key will activate automatic updates of the displayed
To use the file content navigation facilities of \texttt{less} press
To return to automatic updates press the \texttt{F}-key again.
\subsection{\texttt{crontab} on \Dloggerhost}
# this is <crontab>
# ============================================================================
# crontab for netrunner
# ---------------------
10 * * * * /home/dl1/bin/scripts/ 1>>/dev/null 2>&1
50 0 * * * /home/dl1/bin/scripts/ 1>>/dev/null 2>&1
# ----- END OF crontab -----
\subsection{\texttt{/etc/exports} on \Dloggerhost}
/home/dl1/data/DL1 *(ro,root_squash,sync,no_subtree_check)
\subsection{\texttt{/etc/fstab} on \Ddisplayhost}
netrunner:/home/dl1/data/DL1 /data/DL1 nfs defaults,auto,ro 0 0
\subsection{\texttt{/etc/syslog-ng/syslog-ng.conf} on \Dloggerhost}
The following excerpt is from the syslog configuration file on the logger host
The messages from the data acquisition program are written to a local file and
to a second host (the display host \Ddisplayhost) via TCP.
Two local files are maintained.
One of them collects all messages while the other only collects messages that
are not at level info.
# BFO Thies DL1
destination dl1logall { file("/var/log/dl1logall" fsync(yes) perm(0644)); };
destination dl1log { file("/var/log/dl1log" fsync(yes) perm(0644)); };
destination dl1loghugoiv { tcp("" port(2222)); };
filter f_dl1 { match('^DL1logger'); };
filter f_dl1_notice { match('^DL1logger') and not level(info); };
log { source(src); filter(f_dl1); destination(dl1logall); };
log { source(src); filter(f_dl1_notice); destination(dl1log); };
log { source(src); filter(f_dl1); destination(dl1loghugoiv); };
The following definition keeps the \DLlogger\ diagnostics away from the system
filter f_messages { not facility(news, mail) and not filter(f_iptables)
and not filter(f_dl1); };
\subsection{\texttt{/etc/syslog-ng/syslog-ng.conf} on \Ddisplayhost}
On the display host \Ddisplayhost\ the messages from the data acquisition
software are received via TCP and syslog.
They are stored in local files with global read permissions.
# BFO Thies DL1
destination dl1log { file("/var/log/dl1log" fsync(yes) perm(0644)); };
destination dl1logall { file("/var/log/dl1logall" fsync(yes) perm(0644)); };
source vesuv { tcp(port(2222)); };
filter f_dl1 { match('^DL1logger'); };
filter f_dl1_notice { match('^DL1logger') and not level(info); };
log { source(vesuv); filter(f_dl1); destination(dl1logall); };
log { source(vesuv); filter(f_dl1_notice); destination(dl1log); };
With newer versions of \texttt{syslog-ng} you have to use
# BFO Thies DL1
destination dl1log { file("/var/log/dl1log" fsync(yes) perm(0644)); };
destination dl1logall { file("/var/log/dl1logall" fsync(yes) perm(0644)); };
source DL1 { tcp(port(2222)); };
filter f_dl1 { match('^DL1logger' value(MSGHDR)); };
filter f_dl1_notice { match('^DL1logger' value(MSGHDR)) and not level(info); };
log { source(DL1); filter(f_dl1); destination(dl1logall); };
log { source(DL1); filter(f_dl1_notice); destination(dl1log); };
Source code is is not documented here.
You can find the source documentation at
There you will find a documentation for the DL1 software as well as a small
library for serial port access.
Use the instructive source code of well performing software to understand how
the \DTDL\ should be controlled via a serial line RS232 connection.
The program \DLdirect\ supports manual communication with the \DTDL.
Upon invocation it sends one command and displays the response on standard
It might be unwise to use \DLdirect\ while an instance of \DLlogger\ is in
operation on the same serial port.
The program \DLlogger\ is the actual data acquisition program.
% ----- END OF DL1recording.tex -----
......@@ -37,7 +37,7 @@ SUBOBS=$(patsubst,%.o,$(SUBSRC))
all: DL1logger DL1direct
flist: Makefile $(wildcard *.cc *.h *.cfg *.sh) README
flist: Makefile $(wildcard *.cc *.h *.cfg *.sh *.tex *.conf) README
echo $^ | tr ' ' '\n' | sort > $@
.PHONY: edit
......@@ -56,7 +56,7 @@ CPPFLAGS=-I$(LOCINCLUDEDIR) $(FLAGS)
#---------------------------------------------------------------------- $(wildcard *.cc *.h) $(filter-out,$(wildcard *.cc *.h))
ident $^ | egrep '^ *\$$Id:' | sort | uniq | \
awk 'BEGIN{ print "const char* const CVSIDS[]={"; } \
{ print "\"" $$0 "\","; } \
......@@ -106,4 +106,34 @@ tester DL1direct DL1logger: %: %.o $(SUBOBS)
/bin/mv -fv $@ $(LOCBINDIR)
# documentation
# ------------- ; /bin/bash -c "DL1logger -help 2>&1" > $@ ; /bin/bash -c "DL1direct -help 2>&1" > $@
DL1logscript=$(DL1work)/ $(DL1logscript); /bin/cp -vpd $< $@
DL1launchscript=$(DL1work)/ $(DL1launchscript); /bin/cp -vpd $< $@
DL1ntpscript=$(DL1work)/ $(DL1ntpscript); /bin/cp -vpd $< $@
DL1recording.tex: \
$(patsubst %.tex,%,$(wildcard *.tex)): %: %.dvi
%.eps:; ps2epsi $< $@
%.pdf: %.eps; epstopdf $<
%.dvi: %.tex; latex $(patsubst %.tex,%,$<) %.dvi; dvips -ta4 $(patsubst %.dvi,%,$<)
%.pdf: %.tex; pdflatex $(patsubst %.tex,%,$<)
%.x: %.dvi; xdvi $(patsubst %.dvi,%,$<) &
%.psp:; gv $<
%.pdp: %.pdf; acroread $<
# ----- END OF Makefile -----
......@@ -31,11 +31,12 @@
* - 27/11/2008 V1.0 Thomas Forbriger
* - 22/12/2008 V1.1 pass records log messages to data file
* - 25/11/2010 V1.2 g++ was not able to resolve template function split
* ============================================================================
......@@ -56,8 +57,8 @@ namespace dl1 {
const int longtimeout=90;
const int cmaxretries=3;
extern const libtime::TRelativeTime oneday(1);
extern const libtime::TRelativeTime twominutes(0,0,2);
const libtime::TRelativeTime oneday(1);
const libtime::TRelativeTime twominutes(0,0,2);
typedef sff::SFFostream<ts::TIsfftimeseries::Tseries> TIostream;
......@@ -76,9 +77,14 @@ namespace dl1 {
Tlistofstring listofstring(const std::ostringstream& oss,
const std::string& eol)
Tlistofstring los;
tfxx::string::gen_split(los, oss.str(), eol, true);
std::string msg(oss.str());
Tlistofstring v=tfxx::string::split<std::list>(msg, eol, true);
Tlistofstring v=::tfxx::string::split<std::list>(msg, eol, true);
} // Tlistofstring listofstring(const std::ostringstream& oss,
// const std::string& eol)
......@@ -38,7 +38,7 @@
all: doc install
TESTS=tfullwrite tservice tskipdata teststuff tfullread
install: $(LIBS) install-include
......@@ -78,7 +78,7 @@ CFLAGS += -O2 -I${SERVERINCLUDEDIR} $(FLAGS)
# documentation
doc: stuff.f
libsff.doc sff.doc doc: stuff.f fapidxx.f
awk 'BEGIN { flagsff=0; flaglib=0; } \
/^cA/ { flaglib=1; flagsff=0; next; } \
/^cB/ { flaglib=0; flagsff=1; next; } \
......@@ -88,7 +88,7 @@ doc: stuff.f
{ if (flagsff) { print > "sff.doc" ; } \
if (flaglib) { print > "libsff.doc" ; } } \
END { print "c ---- END OF sff.doc ----" > "sff.doc" ; \
print "c ---- END OF libsff.doc ----" > "libsff.doc" ;}' $<
print "c ---- END OF libsff.doc ----" > "libsff.doc" ;}' $^
# C prototypes
......@@ -107,9 +107,10 @@ install-include: $(LOCINCLUDEDIR)/sff.h
gse20.o: gse20.f
stuff.o: stuff.f
fapidxx.o: fapidxx.f
sunfortran.o: sunfortran.c
libsff.a: gse20.o stuff.o sunfortran.o
libsff.a: gse20.o stuff.o sunfortran.o fapidxx.o
%.o: %.f
$(FC) $(FFLAGS) $< -c -o $@
c this is <fapidxx.f>
c ----------------------------------------------------------------------------
c ($Id$)
c Copyright (c) 2010 by Thomas Forbriger (BFO Schiltach)
c some functions/subroutines for compatibility with libfapidxx
c 26/11/2010 V1.0 Thomas Forbriger
c ============================================================================
subroutine sff_select_format(formatid, ierr)
c Select a file format for the next data file to be opened
c input:
c filename name of file
c output:
c ierr error status (ok: ierr=0)
integer ierr
character formatid*(*)
if ('sff') then
print *,'ERROR (libsff, sff_select_format):',
& ' only SFF is supported'
subroutine sff_help_formats()
c Print online help
print *,'This is the Fortran version of libsff'
subroutine sff_help_details()
c Print online help
real sff_encode_libversion, sff_libversion
print *,'The Fortran version of libsff can only handle ',
& 'generic SFF data'
print *,'The library is able to read data of SFF version ',
& sff_libversion()
print *,'The library is encodes data for SFF version ',
& sff_encode_libversion()
c ----- END OF fapidxx.f -----
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment