Gratia Site Collector

Introduction

This topic covers an (optional) installation, configuration and operation of a site-level Gratia collector and reporting services..

Preinstallation

1. HTTP certificate
   You will need to apply for an http certificate for the host you are installing the local Gratia Collector on. If you do not already have one, refer to the Get Grid Certificate? topic of the documentation.

   This http certificate should be copied into an /etc/grid-security/http/ directory as httpcert.pem and httpkey.pem. The directory and certificate files should have the following ownership and permissions:

   drwxr-xr-x  2 daemon root 4096 Nov  5 13:50 /etc/grid-security/http
   -rw-r--r--  1 daemon root 1473 May 29 11:20 /etc/grid-security/http/httpcert.pem
   -rw-------  1 daemon root 1679 May 29 11:20 /etc/grid-security/http/httpkey.pem
   

2. Pacman
   All OSG installations are performed using pacman. This must be installed first. Documentation on pacman can be found in the Installation and update tools section of the documentation. For OSG 1.0.0 we recommend using Pacman 3.26. While older or newer versions may work, you are on your own if you try them.
3. Most installations of OSG services are done as root user
   Paraphrasing from the VDT site, if you install as root the VDT's programs can be made available to all the users on your system. Additionally, the VDT will be able to configure and start some of the grid services you might be installing. Some services can only be run if you are root. Corollary: If you don't install as root, you cannot do these things. Unless your local policy states otherwise, you will want to install the OSG Gratia Collector as root.
4. Consult the Operating Systems topic for supported operating systems.
5. You will need to create an installation directory. This directory will henceforth be referred to as the VDT_LOCATION.

Installation and configuration

Now you're ready to start installing the Gratia Collector and Reporting services.

Shutdown existing installation

If you have an existing Gratia Collector installed and running, you must do a complete shutdown and confirm that all processes associated with it have been stopped. Refer to Gratia Shutdown Guide for details.

Important: Remember you are not shutting down this version, so choose the correct link in that section for the specific version you are replacing. Be sure all services and crons are disabled.

Install the Gratia Collector/Reporting

The installation described here is done as root even though services will not necessarily run as root:

• A few questions regarding trust of the caches from which the software is downloaded will be displayed.
  Please answer y (yes) so that the software can be retrieved.
• Finally, make sure there are no non-standard versions of perl, tcsh, or bash in your $PATH variable.

The following pacman command will install the package:

# pacman -get http://vdt.cs.wisc.edu/vdt_1101_cache:Gratia-Reporting    

Note: This installs both the Gratia-Reporting and Gratia-Collector packages.

See the Pacman section of this guide if you encounter an 'unsupported' platform message.

This will take about 5 minutes to complete, depending upon the system and network connection. During this time you may open a second terminal and watch the progress by monitoring the $VDT_LOCATION/vdt-install.log file.

You should not be asked any other questions during the installation process. The installation should complete with the following message:

Downloading [gratia-reporting-0.28.tar.gz] from [http://vdt.cs.wisc.edu/software//gratia/gratia-reporting/0.28]...

At this point your Gratia collector is fully configured and can being collecting accounting data.

Starting Services

For general information on starting and stopping OSG services, refer to the Starting Services topic of this documentation.

For this Gratia Collector/Reporting installation, the services available are:

 vdt-control --list
Service            | Type   | Desired State
-------------------+--------+--------------
fetch-crl          | cron   | enable
mysql5             | init   | enable
vdt-rotate-logs    | cron   | enable
mysql              | init   | do not enable
apache             | init   | enable
tomcat-55          | init   | enable

So to start all services, you would do:
 vdt-control --on 

A couple very important points regarding Gratia services as deployed by the VDT:

• As you can see above there are 2 MySql services defined. This is an anomaly in the VDT installation that should be corrected in the future. Gratia uses MySql V5. The other mysql entry is related to the apache installation and is not used by Gratia. Since it is marked do not enable, it will never be started.
• With the 2 MySql services installed, the $VDT_LOCATION/setup.sh script sets the ~MySql socket for connecting via the MySql client incorrectly. This does not affect the Gratia services as they connect to the database via a port specified in the $VDT-LOCATION/tomcat/v55/gratia/service-configuration.properties file. However, if you have a need to access the database using the MySql client you need to do so with the --socket argument.

  source $VDT_LOCATION/setup.sh
  mysql --socket /usr/local/osg-collector/vdt-app-data/mysql5/var/mysql.sock --host=localhost -u root -p
  

Secure the MySql database

The MySql database is installed without a root password. You should secure it now.

source $VDT_LOCATION/setup.sh
mysql --socket /usr/local/osg-collector/vdt-app-data/mysql5/var/mysql.sock --host=localhost -u root 
mysql> set password for 'root'@'localhost'=PASSWORD('YOUR_DB_ROOT_PASSWORD');
mysql> quit

You must then configure the Gratia services to use the root password by:

source $VDT_LOCATION/setup.sh
cd $VDT_LOCATION/gratia/etc
printf "YOUR_DB_ROOT_PASSWORD" > mysql5.root.pass
$VDT_LOCATION/vdt/setup/configure_gratia --reporting --services

Validate your installation

Follow the instructions on the Gratia validation page.

Accessing Gratia Administration and Reporting via your browser.

The URLs for the Gratia services are:

• Gratia Administration: http://mycollector.mydomain:myport/gratia-administration/
• Gratia Reporting: http://mycollector.mydomain:myport/gratia-reporting/

Changing your Gratia probes

To start reporting the accounting data to your local/remote Gratia collector, you will need to change the configuration file for your Gratia probes. All references to VDT_LOCATION in this section of the document refer to the installation on your CE node or worker node if you have glexec probes configured.

VERY VERY IMPORTANT: Before you do this, you need to insure that all accounting data for each probe as been reported to the current collector.

On each host that you will be changing to report to your local Gratia collector, you will do the following to insure that all data has been transferred:

1. Verify that there are no probe records in the gratia queue (this would only occur if the current collector has been unavailable). There should be no files in $VDT_LOCATION/gratia/var/tmp/gratiafiles . If there are files here, you will want to wait until these have been process by your Gratia probe cron service entry. When there are no files in the queue, you can proceed.

2. Comment or remove the Gratia probe cron process for your scheduler, named gratia-<scheduler>.
For example, the command to disable a condor probe is vdt-control --off gratia-condor

3. You can now make the changes to ProbeConfig file necessary to report to your local/remote collector. The configuration file for the various probes are shown here:

Probe	$VDT_LOCATION/gratia/probe
Condor	./condor/ProbeConfig
PBS	./pbs/ProbeConfig
LSF	./lsf/ProbeConfig
SGE	./sge/ProbeConfig
gLexec	./glexec/ProbeConfig
metric	./metric/ProbeConfig

Note that the metric probe reports to the official OSG RSV collector at GOC, Indiana. It's highly unlikely that you would want to collect this information locally since there are no graphical reports for this data type.

There are 3 attributes that need to be changed to report to your local/remote collector based on the attributes specified in the service-configuration.properties file:

ProbeConfig attribute	service-configuration.properties attribute
SSLHost	service.secure.connection
SSLRegistrationHost	service.open.connection
SOAPHost	service.open.connection

4. Uncomment your Gratia probe cron service or run vdt-control --on for the relevant probe.

You will now be reporting to your local collector for each of these hosts. You can verify this by going the Gratia administration web service for the collector. You may have to wait for the cron job, which runs every ten minutes.

If you will be forwarding this accounting data to the OSG or ITB Fermilab Gratia Collectors, you will need to set up replication as described in the next section.

Setting up Gratia replication

Gratia has the ability to replicate it’s database contents (actually, the JobUsageRecords) to another instance of Gratia running “out there somewhere”. This replication service can be found on the gratia-administration web service menu.

Assuming your database is the source, and the OSG/ITB Gratia collector at Fermilab is the target:

1. Under the Replication menu, select Job Usage Replication or Metric Replication depending on the accounting data you wish to forward.

2. On the Replication screen, under Remote Host, you will add an entry and then click on the Update button.

If you are forwarding to the central Gratia OSG services, these are the URLs for the respective collectors:

Grid	JobUsageRecord	MetricRecord
ITB	http://gratia.opensciencegrid.org:8881	rsv-itb.grid.iu.edu:8880
OSG	http://gratia.opensciencegrid.org:8880	rsv.grid.iu.edu:8880

3.Then, for your entry, select Test under the Options column on the far right.

Your server will send a dummy transaction to the remote host and let you know whether or not the connection was made.

It should indicate either success or failure. A failure means either your entry was incorrect or the Remote Host is not available. If it is not your entry, contact the Gratia administrator of the remote host or try to access it via your web browser.

4. Now you will select the probes that you want to forward.

Under the Probe Name column, you can specify selective probes, probes for specific VOs or All. Press the Update button to add the entry to the database.

NOTE: If do not specify All, you will need to create a replication entry for each probe or VO you desire to forward. This means repeating steps 2 through 5.

5. Finally, start the replication process by using the Start button under the Options column. Replication should start within a minute or so. You can stop the process with the Stop link (which will also take a minute or so).

From our testing gratia can replicate roughly 2000 records per minute for each active replication entry.

You can view the status of your Replication entries by viewing:

• Running column which indicates naturally if it is running or not
• DBID column indicating the last dbid (key of the JobUsageRecord table) sent
• Row Count column indicating the number of replication entries sent

Complete:
Responsible: ChrisGreen - 5 Nov 2007
Reviewer - date: