Commit 2dea224b authored by thomas.forbriger's avatar thomas.forbriger Committed by thomas.forbriger
Browse files

observer is now under version control

This is a legacy commit from before 2015-05-18.
It may be incomplete as well as inconsistent.
See COPYING.legacy and README.history for details.


SVN Path:     http://gpitrsvn.gpi.uni-karlsruhe.de/repos/TFSoftware/trunk
SVN Revision: 52
SVN UUID:     67feda4a-a26e-11df-9d6e-31afc202ad0c
parents
This is a legacy version of the repository. It may be incomplete as well as
inconsistent. See README.history for details. For the old stock of the
repository copyright and licence conditions apply as specified for versions
commited after 2015-05-18. Use recent versions as a base for new development.
The legacy version is only stored to keep a record of history.
Observer Service Tool (Jan. 2000 Thomas Forbriger IfG Stuttgart)
=====================
observer.cfg
observer.pl*
observer.hourly.pl -> observer.pl*
observer.daily.pl -> observer.pl*
observer.weekly.pl -> observer.pl*
observer.monthly.pl -> observer.pl*
observer_mkdir.pl*
The observer-tool comes with four synonyms (which have to match the settings
in the config file - see below). The typical usage is to call
'observer.hourly.pl' from /etc/cron.hourly, 'observer.daily.pl' from
/etc/cron.daily and so on. The observer script should run under root
privileges as it has to su to user accounts. In the user accounts it looks for
observer scripts matching the given interval and executes them. It checks for
the scripts or programs to match the UID of the user account and to be not
writable by others. The observer tool collects reports from these scripts,
sorts them within one report and logs the report. It may also send the report
via e-mail to an admin.
The config file defines the user accounts to include in the action. It has to
be named $HOME/observer/observer.cfg or the name has to be provided in the
environment variable $OBSERVER_CONFIG.
Settings in observer.cfg:
-------------------------
$OBSERVER_CLIENT:
This is a hash table that defines the user accounts to include in the action
and defines the directories where to look for scripts or programs.
$OBSERVER_OKREPORT:
This hash defines the callable intervals (hourly,...) and which report
levels should be mailed.
$OBSERVER_NOTIFY:
This string variable defines to e-mail addresses of user to send the report
to.
$OBSERVER_LOGDIR:
This string variable defines a log dir for the master logs.
Below the directory defined in $OBSERVER_CLIENT the tool expects a
subdirectory structure like this:
./scripts
./scripts/weekly
./scripts/monthly
./scripts/daily
./scripts/hourly
./log
./log/weekly
./log/monthly
./log/daily
./log/hourly
Where it expects to find executable scripts in ./scripts/hourly, etc. The
client service job scripts or programs have to be executable (file mode bit
set) and must not end with '.bak'.
The scripts and programs are called with the following environment variables
set:
OBS_CLIENT the service client (i.e. the user that owns the script)
OBS_LOG_DIR the proposed directory for user level log files
OBS_SCRIPT_DIR the directory where the script was found
OBS_LOG the full pathname of the proposed log file
OBS_SCRIPT the full pathname of the script
OBS_KEY_STATUS a string indicating a status level ('status:')
OBS_KEY_MESSAGE a string indicating a status message ('message:')
OBS_KEY_OK the ok level index ('O')
OBS_KEY_NOTICE the notice level index ('N')
OBS_KEY_ALERT the alert level index ('A')
OBS_STATUS_OK a full status level string ('status: O')
OBS_STATUS_NOTICE a full status level string ('status: N')
OBS_STATUS_ALERT a full status level string ('status: A')
The status level indices ('ONA') are used to define report level to send via
e-mail in $OBSERVER_OKREPORT.
The client service script or program has to return a success or status message
through stdout. It has to produce two lines in the following scheme:
echo $OBS_STATUS_OK
echo $OBS_KEY_MESSAGE this is an example message...
These values will be used to create a quick status index of all service jobs.
The idea of the three status levels is to give the admin an idea how to react:
OK: the service was executed successfull, no further action is needed
NOTICE: there happend something that is unusual, but is not dangerous;
please have a closer look
ALERT: the service met an error condition, immediate action might be
required!
In addition to the status level the client service may write more lines to
stdout or stderr. All these lines will be catched and appended to the
observer-tool log and report.
It is up to the client service tool to create additional logs and reports. The
preferred name for a report file is given in $OBS_LOG and will be unique to
one run of the observer-tool (i.e. it has a time stamp in the file name that
includes the hour at least).
The observer_mkdir.pl tool
--------------------------
After setting up observer.cfg this tool will create all client directories
that are expected by observer.pl.
---- END OF FILE ----
this is <README.history>
============================================================================
history of this repository
--------------------------
2015-05-18
============================================================================
Brief history
-------------
The contents of this repository previously were part of the TFSoftware
repository. They were extracted into a git repository on 2015-05-18.
----------------------------------------------------------------------------
Milestones
----------
2000-02-09 Initial import to CVS repository
tag: thof_initial_commit
2010-08-08 Conversion to subversion
tag: cvs-svn_conversion
2014-01-01 Check-out with git-svn creating a gatekeeper repository
tag: git-svn_gatekeeper_clone
2015-05-18 Creation of git repository
tags: surgery_*
----- END OF README.history -----
# this is <observer.cfg>
# ----------------------------------------------------------------------------
#
# 17/01/00 by Thomas Forbriger (IfG Stuttgart)
#
# config file for observer.pl
#
# all lines not starting with # will be interpreted by perl - i.e. this file
# will be handled as a perl script
#
# REVISIONS and CHANGES
# 17/01/00 V1.0 Thomas Forbriger
#
# ============================================================================
#
# define client directories as entries in a hash table
# the scripts within the client directories are executed with UID of the
# client
$OBSERVER_CLIENT{'autodrm'} ="/server/users/autodrm/observer";
$OBSERVER_CLIENT{'tester'} ="/lutz/users/tester/tmp/observer";
# define possible interval names by setting report level indices
$OBSERVER_OKREPORT{'hourly'} ="NA";
$OBSERVER_OKREPORT{'daily'} ="ONA";
$OBSERVER_OKREPORT{'weekly'} ="ONA";
$OBSERVER_OKREPORT{'monthly'} ="ONA";
# define the e-mail address of the user to be notified
# use single quotes!
$OBSERVER_NOTIFY='root@localhost';
# define path to observer master log files
$OBSERVER_LOGDIR="/server/users/thof/tmp/observer";
# return true, as this is interpreted as a perl script
return 1;
#
# ----- END OF observer.cfg -----
#!/usr/bin/perl
# this is <observer.pl>
# ----------------------------------------------------------------------------
#
# $Source: /home/tforb/svnbuild/cvssource/CVS/thof/scr/adm/observer/observer.pl,v $
# $Id: observer.pl,v 1.1 2000-02-21 15:06:58 thof Exp $
#
# 17/01/00 by Thomas Forbriger (IfG Stuttgart)
#
# This is the main tool for observing services
#
# Use this as observer.hourly.pl, observer.daily.pl observer.weekly.pl
# and observer.monthly.pl
#
# REVISIONS and CHANGES
# 17/01/00 V0.1 Thomas Forbriger
# 19/01/00 V1.0 first released Version
# 21/01/00 V1.1 introduced report level check for mail
# 25/01/00 V1.2 changed reporting scheme
#
# ============================================================================
#
# we aren't using Sys::Syslog as I did not managed to get any message through
#use Sys::Syslog;
$VERSION="OBSERVER V1.2 central service";
# called program name
# -------------------
$OBSERVER_NAME=$0;
# ----------------------------------------------------------------------------
# static configs
# ==============
#
# default config file
# -------------------
$DEFAULT_CONFIG="$ENV{HOME}/observer/observer.cfg";
#
# syslog priority
$SYSLOG_FACILITY ="user";
$SYSLOG_ALERT ="$SYSLOG_FACILITY.alert";
$SYSLOG_NOTICE ="$SYSLOG_FACILITY.notice";
$SYSLOG_OK ="$SYSLOG_FACILITY.info";
#
# status levels
$STATUS_ALERT ="A";
$STATUS_NOTICE ="N";
$STATUS_OK ="O";
#
# external binaries
# -----------------
$binECHO ="/bin/echo";
$binSU ="/bin/su";
$binBASH ="/bin/bash";
$binLOGGER ="/usr/bin/logger";
$binMAIL ="/usr/bin/mail";
#
# ----------------------------------------------------------------------------
# check environment for config file setting
# -----------------------------------------
if ( -r $ENV{OBSERVER_CONFIG} && -T $ENV{OBSERVER_CONFIG} ) {
$CONFIG_FILE=$ENV{OBSERVER_CONFIG}
} else {
$CONFIG_FILE=$DEFAULT_CONFIG
}
# read config file
# ----------------
do $CONFIG_FILE or die "ERROR: reading config file $CONFIG_FILE: $!\n";
# ============================================================================
#
# define subroutines
# ==================
# log a message (to syslog) - generic code
# ----------------------------------------
sub GENERICLOG {
$loglevel=shift;
open (syslog, "|$binLOGGER -p $loglevel -t $OBSERVER_NAME\\[$$\\]")
or FATAL_ERROR("could not open syslog: $!");
for (@_) { print syslog "$_\n"; }
close syslog;
## for (@_) { syslog('notice', "%s\n", $_) }
}
# log a message (to syslog)
# -------------------------
sub LOG { GENERICLOG($SYSLOG_OK, @_); }
# log a noticeable message (to syslog)
# ------------------------------------
sub NOTICELOG { GENERICLOG($SYSLOG_NOTICE, @_); }
# log an error message (to syslog)
# --------------------------------
sub ERRLOG { GENERICLOG($SYSLOG_ALERT, @_); }
# fatal error condition
# ---------------------
sub FATAL_ERROR {
$title="FATAL ERROR in $OBSERVER_NAME\[$$\]:";
ERRLOG($title, @_);
for ($title, @_) { print stderr "$_\n"; }
$command=sprintf("|%s -s \"FATAL ERROR: %s\" %s",
$binMAIL, $OBSERVER_NAME, $OBSERVER_NOTIFY);
open(MAIL,$command) or die "could not open \'$command\': $!\n";
print MAIL "$title\n\n";
for (@_) { print MAIL "$_\n"; }
close(MAIL);
die "...this is a fatal error...\n";
}
# check file permissions and ownership
# ------------------------------------
#
# We should have a close look at these values as we will change the effective
# UID. If there is any other user haveing write access to file that will be
# executed under foreign privileges this will be a severe security hole.
sub CHECK_FILE {
$client=shift;
$clientuid=shift;
$clientgid=shift;
foreach (@_) {
@statentries=stat("$_");
if ($#statentries < 0) {
FATAL_ERROR("ERROR: could not stat $scriptpath!");
}
$fileuid=$statentries[4];
$filegid=$statentries[5];
$filemode=$statentries[2];
## printf("uid %s\ngid %s\nmode %s\n",
## $fileuid, $filegid, $filemode);
## for (@statentries) { printf("%s, ",$_);} ; printf("\n");
if ($fileuid != $clientuid) {
FATAL_ERROR("ERROR: $client does not own $_!");
}
if ($filegid != $clientgid) {
FATAL_ERROR("ERROR: $_ is not in $client group!");
}
## $val=(($filemode + 0) & 06022);
if ((("$filemode" + 0) & 06022) > 0) {
## printf("mode: %o and-mode: %o\n", $filemode, $val);
FATAL_ERROR((sprintf("ERROR: %s",
$_), sprintf("has insecure mode: (%o)!", $filemode)));
}
}
}
# check directory name
# --------------------
sub CHECK_DIR {
for (@_) {
if ( ! ( -d $_ && -r $_ && -x $_ )) {
FATAL_ERROR("\n directory $_\n is unusable!");
}
}
}
# format status output
# --------------------
sub STATUS_FORM {
return sprintf("%1.1s %10.10s %14.14s %s", $_[0], $_[1], $_[2], $_[3]);
}
# ============================================================================
#
# continue main code
# ==================
#openlog($OBSERVER_NAME, "pid", $SYSLOG_FACILITY);
LOG "entered $VERSION";
# check definitions
# -----------------
unless ( defined %OBSERVER_CLIENT ) {
FATAL_ERROR("config hash \$OBSERVER_CLIENT undefined!"); }
unless ( defined %OBSERVER_OKREPORT ) {
FATAL_ERROR("config hash \$OBSERVER_OKREPORT undefined!"); }
unless ( defined $OBSERVER_NOTIFY ) {
FATAL_ERROR("config variable \$OBSERVER_NOTIFY undefined!"); }
unless ( defined $OBSERVER_LOGDIR ) {
FATAL_ERROR("config variable \$OBSERVER_LOGDIR undefined!"); }
# check log path
# --------------
unless ( -d $OBSERVER_LOGDIR && -x $OBSERVER_LOGDIR && -w $OBSERVER_LOGDIR ) {
FATAL_ERROR("log dir $OBSERVER_LOGDIR is unusable!"); }
# check interval level
# --------------------
$OBSERVER_INTERVAL="NIL";
foreach $interval (keys(%OBSERVER_OKREPORT)) {
if ($OBSERVER_NAME =~ m/$interval/) {
# printf ("I am %s!\n", $interval);
$OBSERVER_INTERVAL=$interval;
}
}
$OBSERVER_INTERVAL =~ m/^NIL$/ and FATAL_ERROR("illegal interval!");
#printf ("Interval is: %s\n", $OBSERVER_INTERVAL);
# ====================
# now scan all clients
# ====================
# initialize report arrays
# ------------------------
@OUTPUT_REPORT=(); # script output
@STATUS_REPORT=(); # status message lines
@ERROR_REPORT=(); # non-fatal errors
# cycle through client list
# =========================
foreach $client (keys(%OBSERVER_CLIENT)) {
# check client
@pwentries=getpwnam($client);
if ($#pwentries < 0) {
FATAL_ERROR("ERROR: $client is unknown to /etc/passwd!");
}
## for (@pwentries) { print "$_\n"; }
$CLIENTUID=$pwentries[2];
$CLIENTGID=$pwentries[3];
## print "$CLIENTUID $CLIENTGID\n";
if ($CLIENTUID == 0) {
NOTICELOG("$client has UID $CLIENTUID!");
$CALLCMD="$binBASH -c ";
} else {
$CALLCMD="$binSU $client -c ";
}
# set directories
$OBSERVER_CLDIR=$OBSERVER_CLIENT{$client};
$OBS_SCRIPT_DIR="$OBSERVER_CLDIR/scripts/$OBSERVER_INTERVAL";
$OBS_LOG_DIR="$OBSERVER_CLDIR/log/$OBSERVER_INTERVAL";
CHECK_DIR("$OBSERVER_CLDIR", "$OBS_SCRIPT_DIR", "$OBS_LOG_DIR");
## printf ("script dir: %s\n",$OBS_SCRIPT_DIR);
# check directory for correct file attributes
CHECK_FILE($client, $CLIENTUID, $CLIENTGID, $OBS_SCRIPT_DIR);
# scan script directory
# ---------------------
opendir (scriptdir, $OBS_SCRIPT_DIR)
or FATAL_ERROR("ERROR: could not open $OBS_SCRIPT_DIR: $!");
@scripts = grep {
(!m/\.bak$/)
&& -f "$OBS_SCRIPT_DIR/$_"
&& -x "$OBS_SCRIPT_DIR/$_"
} readdir(scriptdir);
closedir(scriptdir);
# cycle thorugh scripts
# ---------------------
foreach $script (@scripts) {
# check script for correct file attributes
CHECK_FILE($client, $CLIENTUID, $CLIENTGID, "$OBS_SCRIPT_DIR/$script");
# set up environment
$ENV{OBS_KEY_STATUS} ="status:";
$ENV{OBS_KEY_MESSAGE} ="message:";
$ENV{OBS_KEY_OK} =$STATUS_OK;
$ENV{OBS_KEY_NOTICE} =$STATUS_NOTICE;
$ENV{OBS_KEY_ALERT} =$STATUS_ALERT;
$ENV{OBS_STATUS_OK} ="status: $STATUS_OK";
$ENV{OBS_STATUS_NOTICE}="status: $STATUS_NOTICE";
$ENV{OBS_STATUS_ALERT} ="status: $STATUS_ALERT";
$ENV{OBS_CLIENT} =$client;
$ENV{OBS_SCRIPT} =$script;
$ENV{OBS_SCRIPT_DIR} =$OBS_SCRIPT_DIR;
$ENV{OBS_LOG_DIR} =$OBS_LOG_DIR;
@ltime=localtime;
$ENV{OBS_LOG}=sprintf("%s/%s_%.4d_%.2d_%.2d_%.2d.log",
$OBS_LOG_DIR, $script,
$ltime[5]+1900, $ltime[4]+1, $ltime[3]+1, $ltime[2]);
# initialize report variables
$STATUS_LEVEL="";
$STATUS_MESSAGE="";
# set up command string to be opened
$command=sprintf("%s \"%s/%s\" 2>&1 |",
$CALLCMD, $OBS_SCRIPT_DIR, $script);
LOG "open \'$command\'";
# prepare output log
@thisout=();
## printf("init: %d\n",$#thisout);
# open command and read output from command
open(COMMAND, $command) or
FATAL_ERROR("ERROR: could not open: $command: $!\n");
while (<COMMAND>) {
chomp;
if (/^\s*status:/) {
$STATUS_LEVEL="$'";
$STATUS_LEVEL=~ s/^\s*//;
} elsif (/^\s*message:/) {
$STATUS_MESSAGE="$'";
$STATUS_MESSAGE=~ s/^\s*//;
} else {
push @thisout, $_;
}
## printf("%s(%d): %s\n",$script,$#thisout,$_);
}
close(COMMAND);
# append output to report (in case there is any)
if ($#thisout >= 0) {
push @OUTPUT_REPORT, "$script of $client produced output:";
push @OUTPUT_REPORT, "($command)";
for (@thisout) { push @OUTPUT_REPORT, "> $_"; }
push @OUTPUT_REPORT, " ";
}
## print "status: $STATUS_LEVEL $STATUS_MESSAGE\n";
# check status report of this service
$Read_Level=$STATUS_LEVEL;
$STATUS_LEVEL="";
foreach $key ($STATUS_OK, $STATUS_NOTICE, $STATUS_ALERT) {
if ($Read_Level =~ /^\s*$key\s*$/) { $STATUS_LEVEL=$key; }
}
if (($STATUS_LEVEL =~ m/^\s*$/) || ($STATUS_MESSAGE =~ m/^\s*$/)) {
@thisout=("ERROR in status report:",
"$script of $client returned incomplete status!",
"($command)");
if ($STATUS_LEVEL =~ m/^\s*$/) {
push @thisout,
(sprintf("status level (%s) is not set (regex \"^status:\")",
$Read_Level),
sprintf(" or incorrect (valid keys: %s, %s, %s)",
$STATUS_OK,$STATUS_NOTICE, $STATUS_ALERT));
$STATUS_LEVEL=$STATUS_ALERT;
}
if ($STATUS_MESSAGE =~ m/^\s*$/) {
push @thisout, "status message is not set (regex \"^message:\")";
$STATUS_MESSAGE="**** MISSING ****";
}
ERRLOG(@thisout);
## for (@thisout) { print "$_\n"; };
push @ERROR_REPORT, (@thisout," ");
}
## print "status: $STATUS_LEVEL $STATUS_MESSAGE\n";
# remember status
push @STATUS_REPORT, [($client, $script, $STATUS_LEVEL, $STATUS_MESSAGE)];
} # end of script cycle
} # end of client cycle
# ===================
# now log all reports
# ===================
# set log file names
@ltime=localtime;
$logname=sprintf("%s/%s_%.4d_%.2d_%.2d_%.2d",
$OBSERVER_LOGDIR, $OBSERVER_INTERVAL,
$ltime[5]+1900, $ltime[4]+1, $ltime[3]+1, $ltime[2]);
$LOG_MASTER ="$logname.log";
$LOG_PREVSTATUS ="$OBSERVER_LOGDIR/prevstatus.$OBSERVER_INTERVAL.log";
$LOG_STATUS ="$logname.status";
# create status file from status report
@STATUS_FILE=();
for $stline (@STATUS_REPORT) {
push @STATUS_FILE,
STATUS_FORM($$stline[2], $$stline[0], $$stline[1], $$stline[3]);
};
# compare with previous status report
# -----------------------------------
@PREVIOUS_STATUS=();
@STATUS_DIFF=();
if (-r $LOG_PREVSTATUS) {
if (open(PREVSTATUS, "<$LOG_PREVSTATUS")) {
# read previous
@PREVIOUS_STATUS=(<PREVSTATUS>);
close(PREVSTATUS);
chomp(@PREVIOUS_STATUS);
# compare
foreach $stline (map { s/[\*\?\+]// } sort (@STATUS_FILE)) {
## print "$stline\n";
@matching=grep { /$stline/ } map { s/[\*\?\+]// } (@PREVIOUS_STATUS);
## print "$#matching\n";
if ($#matching < 0) {
push @STATUS_DIFF, $stline;
}
}
if ($#STATUS_DIFF < 0) { @STATUS_DIFF="none"; }
} else {
@STATUS_DIFF=("could not open $LOG_PREVSTATUS: $!");
ERRLOG(@STATUS_DIFF);
} # if open
} else {
@STATUS_DIFF=("There was no previous status log found...");
ERRLOG(@STATUS_DIFF);
} # if -r
$now_time=localtime;
# write status file
# -----------------
if (open(STATUSFILE, ">>$LOG_STATUS")) {
printf STATUSFILE ("%s\n", $now_time);
for (@STATUS_FILE) { print STATUSFILE "$_\n"; }
close(STATUSFILE);
} else {
$error="could not open $LOG_STATUS: $!";
push @ERROR_REPORT, ($error, " ");
ERRLOG($error);
}
# write "previous" status file
# ----------------------------
if (open(STATUSFILE, ">$LOG_PREVSTATUS")) {
for (@STATUS_FILE) { print STATUSFILE "$_\n"; }
close(STATUSFILE);
} else {
$error="could not open $LOG_PREVSTATUS: $!";
push @ERROR_REPORT, ($error, " ");
ERRLOG($error);
}
# prepare full report
# -------------------
@OBSERVER_REPORT=();
@STATUS_HEAD=(STATUS_FORM(("S","user","service","message")),
STATUS_FORM("---","--------------------------",
"-------------------------------",
substr("--------------------------------------------------------", 1,46)));
# find status level
$MASTER_LEVEL="OK";
@STLINES_OK =grep { m/^$STATUS_OK / } sort(@STATUS_FILE);
@STLINES_NOTICE =grep { m/^$STATUS_NOTICE / } sort(@STATUS_FILE);
@STLINES_ALERT =grep { m/^$STATUS_ALERT / } sort(@STATUS_FILE);
@TELL_OK=();
@TELL_NOTICE=();
@TELL_ALERT=();
if ($#STLINES_OK >= 0) {
@TELL_OK=("OK level:", @STATUS_HEAD, @STLINES_OK, " ");
}