Syslog-ng Install

Overview

The following describes a system of two syslog-ng servers: one providing a source of logging data, the second providing a central logging host which can receive logging information from a number of syslog-ng hosts. Here is what was done to setup an initial two-host system:
  • Install syslog-ng on a central logging host (uct3-edge5.uchicago.edu); configure with the standard template for a central logging host.
  • Install syslog-ng on VTB / WS-GRAM site services host; configure with a socket for a specific WS-GRAM/OSG logfiles and forward to the central logging host.
  • Create a bootscripts for these hosts.
  • Launch services on each host by executing the boot scripts.
  • Validate by writing a test message into the service host (into its socket), look for it on the central logging host.

Installing Syslog-ng software

Instructions for an independent syslog-ng instance on any host. Here we focus on installations which do not override or replace the systems native syslog program.
cd /opt/src/
tar xvzf eventlog-0.2.5.tar.gz
tar xvzf syslog-ng-2.0.4.tar.gz
cd eventlog-0.2.5
./configure --prefix=/opt/eventlog
make
make install
#
cd ../syslog-ng-2.0.2
export EVTLOG_CFLAGS="-I/opt/eventlog/include/eventlog"
export EVTLOG_LIBS="-L/opt/eventlog/lib/ -levtlog"
./configure --prefix=/opt/syslog-ng
make
make install

Configuration of the source service host

After installing the syslog-ng software, you need a configuration file and a boot script. In addition, a cron job for the attached python script needs to be added to periodically check the logger processes. Something like log_cron.py -c [path_to_ini_file] -t will do the trick.

This configuration file will go into /opt/syslog-ng/etc/syslog-ng.conf:


options {
   #time_sleep(50);  # polling interval, in ms (helps reduce CPU)
   time_sleep(500);  # polling interval, in ms (make this once per second)
# Note - time_sleep(1000) does not seem to work, should this be 100 ?
   use_fqdn(yes);  # use fully qualified domain names
   ts_format(iso);  # use ISO8601 timestamps
   #
   # for normal load
   flush_lines (10); # number of lines to buffer before writing to disk
   log_fifo_size(100); 
   #
   # for heavy load
   #flush_lines (1000); # number of line to buffer before writing to disk
   #log_fifo_size(1000); 
   #
   stats_freq(3600);  # number of seconds between syslog-ng internal stats events; these are useful
                               # for ensuring syslog-ng is not getting overloaded
};

#
#
source gatekeeper_log { file ("/opt/itb-0.7.0/globus/var/gatekeeper.log" follow-freq(1) flags(no-parse) log_prefix('gatekeeper_log ') ); };
source accounting_log { file ("/opt/itb-0.7.0/globus/var/accounting.log" follow-freq(1) flags(no-parse) log_prefix('accounting_log ') ); };
source error_log { file ("/opt/itb-0.7.0/apache/logs/error_log" follow-freq(1) flags(no-parse) log_prefix('error_log ') ); };
source access_log { file ("/opt/itb-0.7.0/apache/logs/access_log" follow-freq(1) flags(no-parse) log_prefix('access_log ') ); };
source edg_mkgridmap_log { file ("/opt/itb-0.7.0/edg/log/edg-mkgridmap.log" follow-freq(1) flags(no-parse) log_prefix('edg_mkgridmap_log ') ); };
source gridftp_auth_log { file ("/opt/itb-0.7.0/globus/var/log/gridftp-auth.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_auth_log ') ); };
source gridftp_log { file ("/opt/itb-0.7.0/globus/var/log/gridftp.log" follow-freq(1) flags(no-parse) log_prefix('gridftp_log ') ); };
source globus_gatekeeper_log { file ("/opt/itb-0.7.0/globus/var/globus-gatekeeper.log" follow-freq(1) flags(no-parse) log_prefix('globus_gatekeeper_log ') ); };
source container_log { file ("/opt/itb-0.7.0/globus/var/container.log" follow-freq(1) flags(no-parse) log_prefix('container_log ') ); };
source container_real_log { file ("/opt/itb-0.7.0/globus/var/container-real.log" follow-freq(1) flags(no-parse) log_prefix('container_real_log ') ); };
source syslog_ng { internal(); };

#
# define the Forwarding Destination
destination vtb_dst {
        tcp("osg-log.uchicago.edu" port(5145));
};

destination syslog_ng_dst {
  file ("/tmp/syslog-ng.log" perm(0644) );
};
#
# forward sources to destination
# Define what should be forwarded to the destinations
# Definitions
log { source(gatekeeper_log); destination(central_dest); flags(flow-control); };
log { source(accounting_log); destination(central_dest); flags(flow-control); };
log { source(error_log); destination(central_dest); flags(flow-control); };
log { source(access_log); destination(central_dest); flags(flow-control); };
log { source(edg_mkgridmap_log); destination(central_dest); flags(flow-control); };
log { source(gridftp_auth_log); destination(central_dest); flags(flow-control); };
log { source(gridftp_log); destination(central_dest); flags(flow-control); };
log { source(globus_gatekeeper_log); destination(central_dest); flags(flow-control); };
log { source(container_log); destination(central_dest); flags(flow-control); };log { source(container_real_log); destination(central_dest); flags(flow-control); };

# for syslog-ng debugging
log { source(syslog_ng); destination(syslog_ng_dest); };

# This let us verify that syslog-ng is working by using logger (or
# logger-ng) to push things directly to syslog-ng
log { source(test_src); destination(test_dest); };

Boot script for the WS-GRAM service host

Capture and forward globus logfile .
#!/bin/bash
#
# Source function library
. /etc/rc.d/init.d/functions
#
BINDIR=/opt/syslog-ng/sbin
syslog=syslog-ng
config=/opt/syslog-ng/etc/syslog-ng.conf
syslog_pid="/var/run/syslog-ng.pid"
#
# check config and programs
test -s ${config}         || {
    echo 1>&2 "${config} does not exist"
    if test "$1" == "stop" ; then exit 0 ; else exit 6 ; fi
}
test -x ${BINDIR}/$syslog || {
    echo 1>&2 "${BINDIR}/$syslog is not installed"
    if test "$1" == "stop" ; then exit 0 ; else exit 5 ; fi
}
#
case "$1" in
   start)
#       startproc -u nobody -p ${syslog_pid} ${BINDIR}/${syslog} -f $config
       daemon ${BINDIR}/${syslog} "-f $config"  
       echo  "Starting syslog-ng service"
       ;;
   stop)
       echo "Shutting down syslog-ng service"
       # killproc ${syslog_pid} TERM
       killproc ${syslog} TERM 
       ;;
   restart)
       echo "restarting syslog-ng service"
       $0 stop
       $0 start
       ;;
     *)
       echo "Usage: $0 {start|stop|restart}"
       exit 1
       ;;
esac

Ini file settings for log_cron.py script

With the default settings given in the boot script, this file should be located in /usr/local/etc/log_settings.ini:
[Main]
pid_file_location = /var/run/processes.pid
log_socket_directory = /tmp/
logger_binary = /usr/bin/logger

[Log Files]
wsgram = /opt/osg/globus/var/container-real.log
gridftp = /opt/osg/globus/var/gridftp.log
gatekeeper = /opt/osg/globus/var/globus-gatekeeper.log
condor = /opt/condor/local.t2dev-01/log/SchedLog

Remove the condor entries if condor logs should not be sent or if condor is not being used.

Configuration of central logging host

This will be uct3-edge5.uchicago.edu.

This configuration file will go into /opt/syslog-ng/etc/syslog-ng.conf:

options {
   time_sleep(50);  # polling interval, in ms (helps reduce CPU)
   create_dirs(yes);  # create output directories
   use_fqdn(yes);  # use fully qualified domain names
   ts_format(iso);  # use ISO8601 timestamps (syslog-ng 2.0 only)
   #
   # for normal load
   flush_lines (10); # number of line to buffer before writing to disk
   log_fifo_size(100);
   #
   # for heavy load
   #flush_lines (1000); # number of line to buffer before writing to disk
   #log_fifo_size(1000);
   flush_timeout(500); # in ms
   #
   stats_freq(3600);
};
#

#define filters for separating logs
filter gridftp_filter {
    match("gridftp");
};

filter gatekeeper_filter {
    match("gatekeeper");
};

filter condor_filter {
    match("condor");
};

filter wsgram_filter {
    match("container");
};

#
# define the source: any host sending to port 5142
source network_wsgram {
   tcp(port(5142) max-connections(500));
};
source network_gatekeeper {
   tcp(port(5143) max-connections(500));
};
source network_gridftp {
   tcp(port(5144) max-connections(500));
};
source osg_network {
   tcp(port(5145) max-connections(500));
};

source local_wsgram {
   unix-stream("/tmp/container.socket");
};
source local_gatekeeper {
   unix-stream("/tmp/gatekeeper.socket");
};
source local_gridftp {
   unix-stream("/tmp/gridftp.socket");
};
#
#
# Define the destination, automatically creating new directories
#    for each month and new host.
destination wsgram_logs {
     file ("/var/log/vtb/wsgram/$YEAR.$MONTH/vtb.$HOST.log"
           perm(0644) dir_perm(0755) create_dirs(yes)
          template("$ISODATE $HOST $MSG\n") );
};

destination gatekeeper_logs {
     file ("/var/log/vtb/gatekeeper/$YEAR.$MONTH/vtb.$HOST.log"
           perm(0644) dir_perm(0755) create_dirs(yes)
          template("$ISODATE $HOST $MSG\n") );
};

destination gridftp_logs {
     file ("/var/log/vtb/gridftp_logs/$YEAR.$MONTH/vtb.$HOST.log"
           perm(0644) dir_perm(0755) create_dirs(yes)
          template("$ISODATE $HOST $MSG\n") );
};
#
#
destination condor_logs {
     file ("/var/log/vtb/condor_logs/$YEAR.$MONTH/vtb.$HOST.log"
           perm(0644) dir_perm(0755) create_dirs(yes)
          template("$ISODATE $HOST $MSG\n") );
};

log { 
     source(network_wsgram);
     destination(wsgram_logs); 
     flags (flow-control);
};
log { 
     source(network_gatekeeper);
     destination(gatekeeper_logs); 
     flags (flow-control);
};
log { 
     source(network_gridftp);
     destination(gridftp_logs); 
     flags (flow-control);
};
log { 
     source(local_wsgram);
     destination(wsgram_logs); 
     flags (flow-control);
};
log { 
     source(local_gatekeeper);
     destination(gatekeeper_logs); 
     flags (flow-control);
};
log { 
     source(local_gridftp);
     destination(gridftp_logs); 
     flags (flow-control);
};
log { 
     source(osg_network);
     destination(gridftp_logs); 
     filter(gridftp_filter);
     flags (flow-control);
};
log { 
     source(osg_network);
     destination(gatekeeper_logs); 
     filter(gatekeeper_filter);
     flags (flow-control);
};
log { 
     source(osg_network);
     destination(condor_logs); 
     filter(condor_filter);
     flags (flow-control);
};
log { 
     source(osg_network);
     destination(wsgram_logs); 
     filter(wsgram_filter);
     flags (flow-control);
};

Finally, a cron job needs to be added to insure that the tail processes remain running. Something like:

*/10 * * * * /path/to/log_cron.py -c [path_to_ini] -t 
will do.

Bootscript for the central logging host

Note - this is a special central logging host - it also hosts OSG-VTB gatekeeper services, so we may also collecting globus-gatekeeper, gridftp, and container logfiles.
#!/bin/bash
#
# Source function library
. /etc/rc.d/init.d/functions
#
BINDIR=/opt/syslog-ng/sbin
syslog=syslog-ng
config=/opt/syslog-ng/etc/syslog-ng.conf
syslog_pid="/var/run/syslog-ng.pid"
logging_script="/usr/local/bin/log_cron.py"
script_ini="/usr/local/etc/log_settings.ini"
#
# check config and programs
test -s ${config}         || {
    echo 1>&2 "${config} does not exist"
    if test "$1" == "stop" ; then exit 0 ; else exit 6 ; fi
}
test -x ${BINDIR}/$syslog || {
    echo 1>&2 "${BINDIR}/$syslog is not installed"
    if test "$1" == "stop" ; then exit 0 ; else exit 5 ; fi
}
#
case "$1" in
   start)
#       startproc -u nobody -p ${syslog_pid} ${BINDIR}/${syslog} -f $config
       daemon ${BINDIR}/${syslog} "-f $config"  
       echo  "Starting syslog-ng service"
       # create streams for log files here; note dq2 is a tag for the source.
       $logging_script -c $script_ini -s
       ;;
   stop)
       echo "Shutting down syslog-ng service"
       # killproc ${syslog_pid} TERM
       killproc ${syslog} TERM 
       $logging_script -c $script_ini -k
       ;;
   restart)
       echo "restarting syslog-ng service"
       $0 stop
       $0 start
       ;;
     *)
       echo "Usage: $0 {start|stop|restart}"
       exit 1
       ;;
esac

Validation

Write a message into the log sockets on the source host :
[root@uct2-grid1 syslog-ng]# /usr/bin/logger -t test -u /tmp/gatekeeper.socket "this is a test message"

On the destination side (the central logging host), check to see that the message arrived. This directory /working/syslog-ng/dq2/logfiles/2007.03 was created. Then [root@uct3-edge6 2007.03]$grep test dq2.uct2-grid1.uchicago.edu.log yields:

2007-03-27T08:39:24-05:00 uct2-grid1.uchicago.edu test: this is a test message
Message arrived. DONE

Repeating this test with a very long (>400 bytes) message shows that messages are getting truncated. The syslog-ng docs say that the default message length limit is 8192 bytes, but this does not seem to match experiment. I am currently investigating... Charles

References

-- SuchandraThapa - 24 Apr 2007

Topic attachments
I Attachment Action Size Date Who Comment
txttxt log_cron.py.txt manage 6.1 K 08 May 2007 - 21:44 UnknownUser script to check tail processes and start/stop them
Topic revision: r6 - 18 Sep 2007 - 20:52:56 - SuchandraThapa

Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..