The Center for Distributed Petascale Science (CEDPS) SciDAC project is developing a set of tools to aid in Grid troubleshooting. These new tools assume the existence of one or more centralized log file repositories. We assume most sites will want a site repository, and that a subset of the site data will be forwarded to one or more central OSG and/or VO repositories.
The CEDPS project recommends the commonly used, open-source (GPL) tool called syslog-ng
for this purpose. syslog-ng (v1.6) is included with all standard Linux and BSD distributions, but is typically not installed by default. However the new version of syslog-ng (v2.0) is recommended due to support for ISO timestamps, including time zone support
Eventually other CEDPS troubleshooting services will need to be deployed on the central log archive hosts. The requirements for those services will be detailed in a future readiness plan.
syslog-ng can be used to collect existing log files and forward them to one or more log destinations. syslog-ng, unlike standard syslog, includes the following features that make it ideal for this purpose:
- A powerful regular expression (regex)-based filtering mechanism to only forward logs of interest.
- Timezone support
- Fully qualified hostnames
- Automatic reconnect if socket connections fail
- Support for fifos, TCP, and UDP
- Built-in file rotation
- Secure data transport (via stunnel)
Duration: Indefinite, as we expect syslog-ng to be useful to the ITB itself.
For encrypted log file transfers, the standard open source tool stunnel
Otherwise there are no dependencies.
A syslog-ng forwarder should be installed on all hosts running Grid services such as gridFTP, gatekeeper, WS-Gram, VOMS, or Condor-G.
One central collection server per site is needed, and well as one central collection server for OSG.
A moderate amount of disk space is required for the central log file collection hosts (maybe 100 GB?)
One central collection host per site should be identified. CPU and network requirements are minimal, so this does not need to be a dedicated machine.
Note that by default the syslog-ng port of the central collector is open to the world. This can be protected using tcp_wrappers or iptables. The installation instructions for the central collector should include information on optionally configuring one of these mechanisms.
syslog-ng should be packaged with VDT with sample configuration files and installed via pacman.
syslog-ng can be run as a standard user on a non-privileged network port, or it can be installed as a drop-in replacement for the standard system syslog daemon. For OSG we recommend it be installed as user 'nobody' or 'daemon' on a non-privileged network port.
Sample configuration files and boot scripts can be found on the CEDPS wiki
Configuration for vtb sites can be found at VTBSyslogNG
To test the syslog-ng configuration, run something like this:
/usr/bin/logger -t test -u /tmp/gridlogs.socket "this is a test message"
Where "-u" points to the listen socket of your syslog-ng server, (See http://www.cedps.net/wiki/index.php/Syslog-ng
and verify that the test message shows up on the central collection host.
syslog-ng contains a number of performance tuning knobs to help guarantee that it does not drop log messages or overwhelm the central collection host. These will need to be tested on the VTB and ITB. Fortunately syslog-ng reports anytime messages are lost, so it will be fairly easy to monitor whether or not the syslog-ng configuration is working well.
We will need to write test scripts that generate lots of log messages on many hosts to verify that the central collection host can keep up.
Testing must be done to determine the scalability limits for a single receiver in terms of both number of senders and sender rate.
- 14 Mar 2007