Gratia Site Collector
This topic covers an (optional) installation, configuration and operation of a site-level Gratia collector and reporting services..
- HTTP certificate
You will need to apply for an http certificate for the host you are installing the local Gratia Collector on. If you do not already have one, refer to the Get Grid Certificate? topic of the documentation.
This http certificate should be copied into an
/etc/grid-security/http/ directory as
httpkey.pem. The directory and certificate files should have the following ownership and permissions:
drwxr-xr-x 2 daemon root 4096 Nov 5 13:50 /etc/grid-security/http
-rw-r--r-- 1 daemon root 1473 May 29 11:20 /etc/grid-security/http/httpcert.pem
-rw------- 1 daemon root 1679 May 29 11:20 /etc/grid-security/http/httpkey.pem
All ITB installations are performed using pacman. This must be installed first. Documentation on pacman can be found in the
Installation and update tools section of the documentation. For ITB 0.9.2 we recommend using Pacman 3.26. While older or newer versions may work, you are on your own if you try them.
- Most installations of OSG services are done as
Paraphrasing from the VDT site, if you install as root the VDT's programs can be made available to all the users on your system. Additionally, the VDT will be able to configure and start some of the grid services you might be installing. Some services can only be run if you are root. Corollary: If you don't install as root, you cannot do these things. Unless your local policy states otherwise, you will want to install the OSG Gratia Collector as root.
- Consult the Operating Systems topic for supported operating systems.
- You will need to create an installation directory.
This directory will henceforth be referred to as the VDT_LOCATION.
Now you're ready to start installing the Gratia Collector and Reporting services.
Shutdown existing installation
If you have an existing Gratia Collector installed and running, you must
do a complete shutdown and confirm that all processes associated with it have been stopped. Refer to Gratia Shutdown Guide
Remember you are not shutting down this version, so choose the correct link in that section for the specific version you are replacing. Be sure all services and crons are disabled.
Install the Gratia Collector/Reporting
The installation described here is done as root even though services will not necessarily run as root:
- A few questions regarding trust of the caches from which the software is downloaded will be displayed.
Please answer y (yes) so that the software can be retrieved.
- Finally, make sure there are no non-standard versions of perl, tcsh, or bash in your $PATH variable.
The following pacman command will install the package:
# pacman -get http://vdt.cs.wisc.edu/vdt_11098_cache:Gratia-Reporting
This installs both the Gratia-Reporting and Gratia-Collector packages.
See the Pacman section
of this guide if you encounter an 'unsupported' platform message.
This will take about 5 minutes to complete, depending upon the system and network connection. During this time you may open a second terminal and watch the progress by monitoring the
You should not be asked any other questions during the installation process. The installation should complete with the following message:
Downloading [gratia-reporting-0.28.tar.gz] from [http://vdt.cs.wisc.edu/software//gratia/gratia-reporting/0.28]...
At this point your Gratia collector is fully configured and can being collecting accounting data.
For general information on starting and stopping OSG services, refer to the Starting Services
topic of this documentation.
For this Gratia Collector/Reporting installation, the services available are:
Service | Type | Desired State
fetch-crl | cron | enable
mysql5 | init | enable
vdt-rotate-logs | cron | enable
mysql | init | do not enable
apache | init | enable
tomcat-55 | init | enable
So to start all services, you would do:
A couple very important points regarding Gratia services as deployed by the VDT:
The MySql database is installed without a
password. You should secure it now.
mysql --socket /usr/local/osg-collector/vdt-app-data/mysql5/var/mysql.sock --host=localhost -u root
mysql> set password for 'root'@'localhost'=PASSWORD('YOUR_DB_ROOT_PASSWORD');
You must then configure the Gratia services to use the root password by:
printf "YOUR_DB_ROOT_PASSWORD" > mysql5.root.pass
$VDT_LOCATION/vdt/setup/configure_gratia --reporting --services
Follow the instructions on the Gratia validation page
The URLs for the Gratia services are:
To start reporting the accounting data to your local/remote Gratia collector, you will need to change the configuration file for your Gratia probes. All references to VDT_LOCATION in this section of the document refer to the installation on your CE node or worker node if you have glexec probes configured.
VERY VERY IMPORTANT: Before you do this, you need to insure that all accounting data for each probe as been reported to the current collector.
On each host that you will be changing to report to your local Gratia collector, you will do the following to insure that all data has been transferred:
1. Verify that there are no probe records in the gratia queue (this would only occur if the current collector has been unavailable).
There should be no files in
. If there are files here, you will want to wait until these have been process by your Gratia probe cron service entry. When there are no files in the queue, you can proceed.
2. Comment or remove the Gratia probe cron process for your scheduler, named gratia-<scheduler>.
For example, the command to disable a condor probe is
vdt-control --off gratia-condor
3. You can now make the changes to
file necessary to report to your local/remote collector.
The configuration file for the various probes are shown here:
that the metric probe reports to the official OSG RSV collector at GOC, Indiana. It's highly unlikely that you would want to collect this information locally since there are no graphical reports for this data type.
There are 3 attributes that need to be changed to report to your local/remote collector based on the attributes specified in the
4. Uncomment your Gratia probe cron service or run vdt-control --on for the relevant probe.
You will now be reporting to your local collector for each of these hosts. You can verify this by going the Gratia administration web service for the collector. You may have to wait for the cron job, which runs every ten minutes.
If you will be forwarding this accounting data to the OSG or ITB Fermilab Gratia Collectors, you will need to set up replication as described in the next section.
Gratia has the ability to replicate its database contents (actually, the JobUsageRecords) to another instance of Gratia running “out there somewhere”. This replication service can be found on the gratia-administration web service menu.
Assuming your database is the source, and the OSG/ITB Gratia collectors at Fermilab are the target(s):
1. Under the
Job Usage Replication
depending on the accounting data you wish to forward.
2. On the
, you will add an entry and then click on the
If you are forwarding to the central Gratia OSG services, these are the URLs for the respective collectors:
You can select specific probes below to ensure that information is forwarded to the right place.
3.Then, for your entry, select
column on the far right.
Your server will send a dummy transaction to the remote host and let you know whether or not the connection was made.
It should indicate either success or failure. A failure means either your entry was incorrect or the
is not available (an error message including the number "503" indicates the latter and is fine: try again later). If a non-transient problem is indicated and you are sure the URL is correct., try to access the remote via your web browser and contact the Gratia administrator of the remote host if you are unable to resolve the problem locally.
4. Now you will select the probes that you want to forward.
column, you can specify selective probes, probes for specific VOs or All. Press the
button to add the entry to the database.
If do not specify
, you will need to create a replication entry for each probe or VO you desire to forward. This means repeating steps 2 through 5.
5. Finally, start the replication process by using the
button under the
column. Replication should start within a minute or so. You can stop the process with the
link (which will also take a minute or so).
Please also consider replicating ProbeDetails records (same procedure as above but select the ProbeDetails replication page): this enables the OSG collectors to receive probe version information and also to distinguish between a probe that is not reporting and a probe that merely has nothing to report.
From our testing gratia can replicate roughly 2000 records per minute.
You can view the status of your Replication entries by viewing:
Running column which indicates naturally if it is running or not
DBID column indicating the last
dbid (key of the JobUsageRecord table) sent
Row Count column indicating the number of replication entries sent
- 5 Nov 2007
Reviewer - date: