Gratia Transfer Probe


Included topic: Gratia Transfer Probe

The BeStMan server is using GridFTP server to transfer files to/from any Posix compliant file system. Gratia GridFTP transfer probe generates accounting information about these transfers by parsing the Globus GridFTP server logs. This document describes the Gratia GridFTP transfer probe configuration procedure. This procedure must be repeated once for every GridFTP server in the SE.


Configuring Gratia GridFTP transfer probe

Gratia GridFTP transfer probe is installed simultaneously with BeStMan or standalone GridFTP server. It is not enabled by default. In order to see what is installed from ITB:BeStMan execute the following commands:

cd <VDT_LOCATION>
. setup.sh
vdt-control --list
[root@fg0x1 itb_bestman]# vdt-control --list
Service            | Type   | Desired State
-------------------+--------+--------------
fetch-crl          | cron   | enable
vdt-rotate-logs    | cron   | enable
vdt-update-certs   | cron   | enable
gsiftp             | inetd  | enable
gratia-gridftp-tran| cron   | do not enable
bestman            | init   | enable
edg-mkgridmap      | cron   | do not enable
gums-host-cron     | cron   | do not enable

To configure Gratia GridFTP transfer probe do the following:

cd <VDT_LOCATION>
. setup.sh
$VDT_LOCATION/vdt/setup/configure_gratia  \
 --probe gridftp-transfer \
 --report-to <gratia_host:gratia_port> \
 --probe-cron --site-name <SiteName>

Where gratia_host is FQDN of Gratia collector, gratia_port is Gratia Collector port and SiteName is your site name. For ITB collector use the following values:
gratia_host gratia-osg-itb.opensciencegrid.org
gratia_port 8881

For OSG collector use the following values:
gratia_host gratia-osg-transfer.opensciencegrid.org
gratia_port 8881

If you want to do additional changes in configuration, eg change location of gridftp logs or name of the Grid (default is OSG) you can edit ProbConfig file located in $VDT_LOCATION/gratia/probe/gridftp-transfer directory.

Creating osg-user-vo-map

Please read this page about files that are required to accurately collect grid resource usage and metrics by VO for transfer submitted using grid proxies or where voms proxy information is not available.

Using GUMS based authorization mode

In order to enable generation of osg-user-vo-map from GUMS do the following:
cd <VDT_LOCATION>
. setup.sh
vdt-control -enable gums-host-cron
vdt-control -on gums-host-cron
touch $VDT_LOCATION/monitoring/osg-attributes.conf
$VDT_LOCATION/gums/scripts/gums-host-cron

Using Gridmap based authorization mode

In order to enable generation of gridmap-file and osg-user-vo-map by using the edg-mkgridmap cron process to get information form VOMS servers do the following:

cd <VDT_LOCATION>
. setup.sh
vdt-control -enable edg-mkgridmap
vdt-control -on edg-mkgridmap
$VDT_LOCATION/edg/sbin/edg-mkgridmap 

Enable Gratia GridFTP Transfer probe

You will now need to enable Gratia GridFTP transfer probe:

cd <VDT_LOCATION>
. setup.sh
vdt-control --enable gratia-gridftp-transfer

Starting/Stopping Gratia GridFTP transfer probe

  1. Gratia GridFTP probe installed on BeStMan or GridFTP nodes could be started together with all other services by issuing vdt-control --on command. If you want to start it separately do the following:
    cd <VDT_LOCATION>
    . setup.sh
    vdt-control --on gratia-gridftp-transfer
    
  2. Gratia GridFTP probe installed on BeStMan or GridFTP nodes could be stopped together with all other services by issuing vdt-control --off command. If you want to stop it separately do the following:
cd <VDT_LOCATION>
. setup.sh
vdt-control --off gratia-gridftp-transfer

Log files and configuration location

You could find log and configuration files related to probe at the following location on a relevant node:

Module Name Configuration files Log files
Gratia GridFTP probe $VDT_LOCATION/gratia/probe/gridftp-transfer/ProbeConfig $VDT_LOCATION/gratia/var/logs

Sanity Check

If you turned on GridFTP Gratia transfer probes you should be able to see the accounting information by accessing your Gratia collector. Keep in mind that probe collection is executed by a cron job, so check the time the cron job will be executed:

crontab -l
5,15,25,35,45,55 * * * * <VDT_LOCATION>/gratia/probe/gridftp-transfer/gridftp-transfer_meter.cron.sh > <VDT_LOCATION>gratia/var/logs/gratia-probe-gridftp-transfer.log 2>&1

To access the information, go to http://<gratia_host>:<gratia_port>/gratia-reporting/, click on "Custom SQL Query" on the left site menu frame, enter the following query into provided text box:

select * from MasterTransferSummary where ProbeName like 'gridftp-transfer:<your_host_name>';
click on "Execute Query" and you will see the total number of transfer per user. To get more detailed information you can execute the following query:
select j.dbid, j.ResourceType ,j.LocalJobId,LocalUserId,j.CommonName,j.KeyInfoContent,j.Status,j.StartTime,
         j.EndTime,j.SubmitHost,m.ReportedSiteName,m.Grid, n.Value,n.StorageUnit 
         from JobUsageRecord j, JobUsageRecord_Meta m, Network n  
         where j.dbid=m.dbid and j.dbid=n.dbid and m.ProbeName like 'gridftp-transfer:<your_host_name>'  order by dbid;

To check ITB Gratia collector click here ITB Gratia.

To check OSG Gratia collector click here OSG Gratia.

Known issues

More information


Complete: 1
Responsible: TanyaLevshina - 18 Feb 2009
Reviewer - date:

Topic revision: r5 - 16 Apr 2009 - 13:37:45 - TanyaLevshina
 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..