BeStMan Gateway

Please note: This documentation is for OSG 1.2. While we still provide critical security updates for OSG Software 1.2, we recommend you use OSG Software 3 for any new or updated installations. We are considering May 31, 2013 as possible OSG 1.2 End of Life (EOL).

ReleaseDocumentation
BestmanGateway
Reviewed Passed
by AlexSim
Test Passed
by MarcoMambelli
Released
by DouglasStrain

About this Document

hand This document is for Storage System Administrators. It contains the installation procedure for a Berkeley Storage Manager (BeStMan) in gateway-mode. This document applies to the latest release OSG-1.2.28 .

Conventions used in this document:

A User Command Line is illustrated by a green box that displays a prompt:

  [user@bestman /opt/osg-1.2.32-bestman]$

A Root Command Line is illustrated by a red box that displays the root prompt:

  [root@bestman /opt/osg-1.2.32-bestman]$

Lines in a file are illustrated by a yellow box that displays the desired lines in a file:

priorities=1

Introduction

Berkeley Storage Manager (BeStMan) is a generic SRM v2.2 load balancing front-end for transfer servers and disk-based storage systems. It was developed by the Scientific Data Management Group at Lawrence Berkeley National Laboratory.

BeStMan works on top of any disk-based POSIX-compliant file-systems. It is known to work on file systems such as NFS, GPFS, PVFS2, GFS, Ibrix, HFS+, Hadoop, XrootdFS and Lustre and integrates with various file transfer services, such as gsiftp, http, https, bbftp and ftp.

BeStMan-Gateway supports subset of SRM v2.2 on any existing file system without internal queuing or space management. This is a main difference between BeStMan-Gateway and BeStMan -fullmode.

For general information on storage software architecture, implementations and use, please read on storage infrastructure software. For information on planning, installing and validating storage software see here.

Engineering Considerations

BeStMan requires at least one node with following components to be installed:

bestman_gateway_arch.jpeg

Please answer following questions before you proceed with installation and configuration of BeStMan-gateway-mode storage element:

Q. What authorization mechanism do you prefer?
Decide between a grid-mapfile or a GUMS server for authorization.
We recommend to use GUMS as the most flexible solution; most large sites use GUMS.

Q. How many GridFTP servers you will need?
Choose to run multiple GridFTP servers for load balancing and better performance. We recommend to install additional GridFTP servers if your Storage Element:
  • is serving data to more than 250 cores for VOs that use storage heavily (e.g. CMS, ATLAS, CDF, and D0)
  • is managing more than 50 TB of disk space
  • has more than 1Gbps bandwidth: plan on at least one GridFTP server for each 4Gbps of available bandwidth to maximize throughput

Q. Do you need to enable Gratia gridftp-transfer probes?
The Gratia gridftp-transfer probes provide OSG storage statistics for accounting purposes. More details can be found at the Gratia Home Page. The reports include the source and destination of transfers, certificate subject of transfer initiator, as well as the size and status of the transferred file.
The probe needs to be installed on every GridFTP server which may be different from your BeStMan server.

Q. Do you need to support static space tokens?
BeStMan-Gateway supports pre-defined, static space tokens that could be included in configuration. It doesn't keep track of the contents in the space with the static space tokens, nor enforce allocations. If you want to partition your storage space and have a “designated” space for some VOs or users, you can choose to use space tokens. You will have to provide the names and descriptions of the tokens as well as the size of the area.

How to get Help?

Requirements

  1. all procedures in this document require root privileges
  2. the installation procedure requires Pacman version >= 3.28 to be installed
  3. you will need at least 1 server to install the software
  4. the extended internet daemon xinetd must be installed and running on the server
  5. the operation of BeStMan requires a valid service certificate. If you are planning to support access to your SE by LCG-Utils tools this certificate must be the copy of the host certificate (see this section for details). Otherwise use a valid service certificate?.
  6. the service certificate must be owned by the Unix user that is running BeStMan
  7. the firewall must allow incoming connections to the BeStMan ports (default:10080,10443)
  8. the firewall must allow incoming connections to the GridFTP port (default 2811)
  9. outgoing connections must be allowed from high ports ( typically in range 32769-65535 ). We recommend to consult the Firewall Guide if you install the GridFTP server for the first time.
  10. Grid users will authenticate with your site using their grid certificates. The users have to be authorized in order to access your SE. You'll need to determine the authorization mode you wish to use before you can proceed with installation(see for details).
  11. the server must have a fully qualified domain name and a valid host certificate? installed in /etc/grid-security

BeStMan Installation Procedure

The installation procedure consists of the following steps:

  1. verify the operation of xinetd
  2. verify the service certificate for BeStMan
  3. create an installation directory
  4. use Pacman to install BeStMan
  5. install the CA Certificates and the Certificate Revocation List
  6. execute the post installation script

Verify Xinetd Status

The GridFTP service will be started by the xinetd service on connection requests. Please verify that xinetd is running:

[root@bestman /opt/osg-1.2.32-bestman]$ /etc/init.d/xinetd status
xinetd (pid xxxx) is running...

Otherwise consult the documentation provided with your operating system to find out how to install xinetd.

Verify the BeStMan Service Certificate

By default BeStMan is using the certificate and key installed in /etc/grid-security/http/httpcert.pem and /etc/grid-security/http/httpkey.pem respectively. If you don't require support for LCG-Utils simply proceed to request and install a service certificate?.

ALERT! WARNING!
In order to handle requests from LCG-Utils, BeStMan must be configured to use a copy of the host certificate as its service certificate. This is mandatory for all BeStMan servers supporting the ATLAS experiment. Also some grid users are using lcg-utils to access the SE.

There are two ways to solve this problem:

  1. If you don't have a service certificate installed in /etc/grid-security/http, simply copy the host certificate to the location of the service certificate:

[root@bestman ~]$mkdir -p /etc/grid-security/http
[root@bestman ~]$cp /etc/grid-security/hostcert.pem /etc/grid-security/http/httpcert.pem
[root@bestman ~]$cp /etc/grid-security/hostkey.pem /etc/grid-security/http/httpkey.pem
[root@bestman ~]$chown -R user.group /etc/grid-security/http
[root@bestman ~]$chmod 600 /etc/grid-security/http/httpcert.pem /etc/grid-security/http/httpkey.pem

  1. If you have a service certificate installed in /etc/grid-security/http already, create a new sub-directory and copy the host certificate to this location:

[root@bestman ~]$mkdir -p /etc/grid-security/bestmancert
[root@bestman ~]$cp /etc/grid-security/hostcert.pem /etc/grid-security/bestmancert/bestmancert.pem
[root@bestman ~]$cp /etc/grid-security/hostkey.pem /etc/grid-security/bestmancert/bestmankey.pem
[root@bestman ~]$chown -R user.group /etc/grid-security/bestmancert
[root@bestman ~]$chmod 600 /etc/grid-security/bestmancert/bestmancert.pem  /etc/grid-security/bestmancert/bestmankey.pem

HELP NOTE
user and group must correspond to the Unix account used to run BeStMan!

Create the Installation Directory

Create an installation directory and change into it. Make sure the directory is world readable if the installation is to be shared by grid users:

[root@bestman ~]$ mkdir -p /opt/osg-1.2.32-bestman
[root@bestman ~]$ cd /opt/osg-1.2.32-bestman

ALERT! WARNING!
Please do not use a system directory like /opt or /usr for the installation directory. The installation routine will create many sub-directories in the main directory.

Use Pacman to Install BeStMan

Next, we will use Pacman to install BeStMan into the current working directory (/opt/osg-1.2.32-bestman).

If you are using the Grid User Management System (GUMS) as a Grid Identity Mapping Service, you will need to define the $VDT_GUMS_HOST environment variable first. Otherwise skip this step.

[root@bestman /opt/osg-1.2.32-bestman]$ export VDT_GUMS_HOST=<your GUMS hostname>

To download the BeStMan package from the http://software.grid.iu.edu/osg-1.2 cache execute the pacman command. Pacman will ask whether you want to "trust the caches and accept the license", answer yall and y to install the BeStMan package:

[root@bestman /opt/osg-1.2.32-bestman]$ pacman -get http://software.grid.iu.edu/osg-1.2:Bestman

[root@bestman /opt/osg-1.2.32-bestman]$ pacman -get http://software.grid.iu.edu/osg-1.2:Bestman
Do you want to add [http://software-itb.grid.iu.edu/osg-1.2] to [trusted.caches]? (y/n/yall): yall
Beginning VDT prerequisite checking script vdt-common/vdt-prereq-check...        

All prerequisite checks are satisfied.
                                             

========== IMPORTANT ==========
Most of the software installed by the VDT *will not work* until you install
certificates.  To complete your CA certificate installation, see the notes
in the post-install/README file.

Please check the installation log file /opt/osg-1.2.32-bestman/vdt-install.log for errors if the installation failed. Otherwise proceed.

Update the Environment

Depending on your shell update your environment by sourcing /opt/osg-1.2.32-bestman/setup.sh or /opt/osg-1.2.32-bestman/setup.csh:

[root@bestman /opt/osg-1.2.32-bestman]$ . /opt/osg-1.2.32-bestman/setup.sh

Depending on your preference you might want to optionally include the setup script in your system or user profile.

Install a Certificate Authority Package

You will need to install a Certificate Authority package. For more information and options see this document.

Local Installation of the CA Certificates

This local installation of the Certificate Authority Package is preferably be used by grid users without root privileges or if the CA certificates will not be shared by other VDT installations on the same host.

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-ca-manage setupca --location local --url osg
Setting CA Certificates for VDT installation at '/opt/osg-1.2.32-bestman'

Setup completed successfully.

After a successful installation the certificates will be installed in ($VDT_LOCATION/globus/share/certificates, /opt/osg-1.2.32-bestman/globus/share/certificates in this example).

Enable Updates of the CA Certificates

CA certificates have a limited lifetime and will expire. To keep the installed certificates current it is necessary to update them automatically using the vdt-update-certs provided by the Virtual Data Toolkit:

To enable the service use:

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --enable vdt-update-certs
running 'vdt-register-service --name vdt-update-certs --enable'... ok

Enable Updates of the Certificate Revocation List

The Certificate Revocation List lists certificates that have been temporarily or permanently revoked. To keep the CRL current it is necessary to update it automatically using fetch-crl provided by the Virtual Data Toolkit:

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --enable fetch-crl
running 'vdt-register-service --name vdt-update-certs --enable'... ok

Run the Post-Install Script

At last run the vdt-post-install script to finish the installation:

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-post-install
Starting...
Configuring PRIMA... Done.
Configuring EDG-Make-Gridmap... Done.
Done.

This completes the installation of the BeStMan server. Move to the next section to configure the BeStMan-gateway-mode and GridFTP server. You may also choose to configure the Gratia transfer probe at this time (see this section for detailed explanation).

If you want to install additional GridFTP servers on the different nodes, please follow this document for GridFTP server installation. See this section for reasoning.

BeStMan-gateway Configuration

You will need to configure BeStMan-Gateway first in order to enable it as a service. BeStMan is recommended to run under non-root account.

The simplest configuration example is shown below. It utilizes all the default values including the default user (in this configuration BeStMan runs as user daemon) and the default path to certificate and key ( /etc/grid-security/http/httpcert.pem, /etc/grid-security/http/httpkey.pem) that should have the right set of permission and belong to user daemon. See Requirements for details.

[root@bestman /opt/osg-1.2.32-bestman]$ $VDT_LOCATION/vdt/setup/configure_bestman --server y --enable-gateway

This command will install BeStMan that will be

  • running by user daemon,
  • using certificate /etc/grid-security/http/httpcert.pem and certificate key /etc/grid-security/http/httpkey.pem
  • listening on secure port 10443 and public port 10080
  • utilizing gridftp server running on the same host on 2811 port
  • using grid-mapfile for authorization
warning This configuration command doesn't specify any storage area for BeStMan , so the user will be able to write files only in the directories where she can access to (e.g /tmp or under user's home directory if it exists on the node).

If you don't want to try more complex configuration, you may skip the remaining of this and proceed to the next sub-section about the sudoers file.

The more complex installation example is shown below:

The example below shows how to configure BeStMan in gateway-mode, enable GUMS and space token usage. If you would like to use pre-defined, static space tokens you need to provide a list of space token names, description and size of space allocated for each token. Keep in mind that in gateway-mode, BeStMan is not managing the space.

[root@bestman /opt/osg-1.2.32-bestman]$ $VDT_LOCATION/vdt/setup/configure_bestman --server y \
--user user \
--cert service_cert \
--key service_key \
--http-port public_port \
--https-port secure_port \
--gums-host GUMS hostname \
--gums-port GUMS port \
--enable-gateway \
--with-allowed-paths allowed_dir_list \
--with-blocked-paths blocked_dir_list \
--with-tokens-list "TOKEN_1_NAME[desc:TOKEN_1_DESC][TOKEN_1_SIZE_GB];TOKEN_2_NAME[desc:TOKEN_2_DESC][TOKEN_2_SIZE]" \
--with-transfer-servers GridFTP_servers_list

Argument Value Default Value Comment
user_name daemon name of the non-privileged user that runs BeStMan server process
service_cert /etc/grid-security/http/httpcert.pem is a path to service certificate
service_key /etc/grid-security/http/httpkey.pem is a path to service certificate private key
public_port 10080 BeStMan public port
secure_port 10443 BeStMan private port warning Please, make sure that these two ports are open if you have a firewall on your node
GUMS_hostname localhost is the FQDN name of GUMS server
GUMS_port 8443 is the port of GUMS server
GridFTP_servers_list localhost is a list FQDN of your GridFTP servers, separated by ; . e.g. “gsiftp://host1.domain.tld;gsiftp://host2.domain.tld”
low_port,high_port N/A the open ports range that is allowed for all outbound globus connections for gridftp (see also how to deal with firewall in in this section)
allowed_dir_list   list of directories, separated by semicolon, accessible to users
blocked_dir_list /;/etc;/var list of directories separated by semicolon, non-accessible to users (default are "") . One of the --with-allowed-paths or --with-blocked-paths options should be used for storage access policy.
token_list N/A token list format: token_name[KEY:VALUE][token_size_in_GB]

Where token list format:

  • KEY = desc, owner, retention, latency, path, usedBytesCommand. All KEY:VALUE pairs are optional
  • desc = in ATLAS experiment, desc value needs to be the same as the space token name as exampled below.
  • retention available values = CUSTODIAL, OUTPUT, REPLICA. Normally, REPLICA and CUSTODIAL are used.
  • latency available values = ONLINE, NEARLINE
  • usedBytesCommand = e.g. some custom script or "du -s -b". Its output shall have the available bytes as the first value
  • multiple token names are separated by semi-colon

For example: "USATLASDATA1[desc:USATLASDATA1][owner:atlas][retention:REPLICA][latency:ONLINE][path:/project/usatlas/data][usedBytesCommand:/usr/bin/du -s -b][120]"

If you want to use grid-mapfile for user authentication and authorization do not specify the following options:

--gums-host
--gums-port

If you do not want to use pre-defined, static space tokens, do not specify the following options:

--with-tokens-list

If you are running your BeStMan-Gateway on the node that doesn’t have an access to your file system, you will have to modify the following attributes in $VDT_LOCATION/bestman/conf/bestman.rc configuration file:

 
checkSizeWithFS=false 
checkSizeWithGsiftp=true 
These options will allow file system access through GridFTP. However, this requires full GSI delegation from the clients.

Modify /etc/sudoers

# Comment out this line, if it is in your =/etc/sudoers= file (RHEL5+)
#Defaults    requiretty

Cmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv, /bin/ls 
Runas_Alias SRM_USR = ALL, !root 
<user_name> ALL=(SRM_USR) NOPASSWD: SRM_CMD
 

HELP NOTE
user_name refers to the Unix account running the BeStMan server process.

GridFTP Configuration

GridFTP server comes with BeStMan installation and have to be configured if used with it GUMS authorization method.

If you are using GUMS authorization method, please, copy the following two files from $VDT_LOCATION/post-install to /etc/grid-security. Otherwise go to the next step:

[root@bestman /opt/osg-1.2.32-bestman]$cp $VDT_LOCATION/post-install/prima-authz.conf /etc/grid-security 
[root@bestman /opt/osg-1.2.32-bestman]$cp $VDT_LOCATION/post-install/gsi-authz.conf /etc/grid-security 

If you have a firewall, the open port range for the gridftp should be properly set. In order to do so you will have to modify vdt-local-setup.sh and vdt-local-setup.csh files


#edit $VDT_LOCATION/vdt/etc/vdt-local-setup.sh 
GLOBUS_TCP_SOURCE_RANGE=low_port,high_port
GLOBUS_TCP_PORT_RANGE=low_port,high_port
export GLOBUS_TCP_SOURCE_RANGE
export GLOBUS_TCP_PORT_RANGE 

Where low_port, high_port controls all outbound globus connections for gridftp (e.g 40000,49150). You should select the same range of ports you have selected during BeStMan-gateway-mode configuration with --globus-tcp-port-range low_port,high_port option.

You can have multiple installation of GridFTP servers located on the nodes you have specified in BeStMan-gateway mode configuration (see separate GridFTP installation if you want to install it as a stand-alone server).

You will need to make sure that users have write permissions in to the storage area.

Gratia GridFTP Transfer Probe Configuration

To enable and configure the Gratia GrdFTP transfer probe for the GridFTP server that comes with BeStMan follow the instructions in Preparing, Installing and Validating Gratia transfer probe

Enable Services

Before a service can be activated it needs to be enabled. You can list the status of registered services to see if a service is enabled or disabled.

To enable a registered service use vdt-control:

[user@bestman /opt/osg-1.2.32-bestman]$ vdt-control --enable vdt-rotate-logs gsiftp gratia-gridftp-transfer (gums-host-cron|edg-mkgridmap)

HELP NOTE
Please only choose gums-host-cron if you are using GUMS. If you want to generate local dynamic gridmap file choose edg-mkgridmap service instead. If you are using existing gridmap file that has been already installed on your system, do not specify neither gums-host-cron nor edg-mkgridmap services.

If you have decided to use grid-map file for authorization and would like that it has been automatically generated for you do the following after you have enabled edg-mkgridmap service :

[root@bestman /opt/osg-1.2.32-bestman]$ edg/sbin/edg-mkgridmap

Service Activation

Use vdt-control to activate registered services. This will:

  • add entries to crontab for cron services
  • add control scripts to /etc/init.d for init services
  • start new init services
  • configure the xinet daemon for xinet services

Unprivileged users must provide the --non-root argument to vdt-control to install cron services. All other services require root privileges.

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --on 

vdt-control will fail to activate any service that is already provided by the operating system. In this case you may force the activation of the new service provided by the Virtual Data Toolkit:

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --force --on 

Another reason for vdt-control to fail to activate a service may be that the service was previously installed by another installation of the Virtual Data Toolkit which has not been deactivated yet. In this case you must force the deactivation of the existing service before you continue to install the new service:

[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --force --off 
[root@bestman /opt/osg-1.2.32-bestman]$ vdt-control --on 

Validation of Service Operation

Site registration and daily monitoring

Once you have your SE setup and configured, you can register it with the LBNL SRM monitoring system. This will run daily tests against your SE and the results can be viewed here.

Self-testing with srm clients

In order to verify that the system is functional you will need to have access to srm client commands, be able to create a proxy certificate (grid-proxy-init or voms_proxy_init command) and have access to your certificate and private key.

There are multiple ways of doing this.

Preparing to run srm-client command on the BeStMan node

If you don't have an access to the node where OSG Client is installed you can still test BeStMan.
  • Login on the BeStMan node as "yourself"
  • Setup location of BeStMan installation
source /opt/osg-1.2.32-bestman/setup.sh

Make sure that you have access to your certificate and private key on that node. You will need it to create a proxy certificate. Execute grid-proxy-init :

grid-proxy-init
Your identity: .....
Enter GRID pass phrase for this identity:

You will need to add the path to all the srm client commands to your PATH environment variable.

export PATH=$PATH:$VDT_LOCATION/bestman/bin

Preparing to run srm-client command from the different node

You may have an access to the node where OSG client is already installed and where you have your certificate and key. If you want to install OSG Client use instructions provided in in that installation Guide. Then generate a proxy certificate by issuing grid_proxy_init or voms_proxy_init:

source VDT_LOCATION/setup.sh
grid-proxy-init
Your identity: .....
Enter GRID pass phrase for this identity:

or

source VDT_LOCATION/setup.sh
voms-proxy-init -voms VO
Enter GRID pass phrase for this identity:
Where VDT_LOCATION is the directory where the client is installed and VO name of your Virtual Organization.

Executing SRM-client commands

After you manage to get your personal proxy certificate created and access to srm client commands, you can verify BeStMan server installation:

Execute srm-ping:

 srm-ping srm://BeStMan_host:secure_port/srm/v2/server 
########################################### 
SRM_HOME is /usr/local/osg-client/srm-client-lbnl 
JAVA_HOME is /usr/local/osg-client/jdk1.5 X509_CERT_DIR = 
/usr/local/osg-client/globus/TRUSTED_CA 
GSI_DAEMON_TRUSTED_CA_DIR = /usr/local/osg-client/globus/TRUSTED_CA 
########################################### 

SRM-CLIENT: got remote srm object 
 
SRM-PING: Thu Sep 18 11:55:50 CDT 2008 Calling SrmPing Request... 
Ping versionInfo=v2.2 
Extra information 
        Key=backend_type 
        Value=BeStMan 
        Key=backend_version 
        Value=2.2.1.1 
        Key=GatewayMode 
        Value=Enabled 
        Key=gsiftpTxfServers 
        Value=gsiftp://osg-ress-2.fnal.gov 
        Key=clientDN 
        Value=/DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821 
        Key=localIDMapped 
        Value=fnalgrid 
        Key=staticToken(0) 
        Value=DISK1 desc=DATA1 size=1073741824 
        Key=staticToken(1) 
        Value=DISK2 desc=DATA2 size=2147483648
Please check that your gumsIDMapped is not null. If this is the case you have probably misconfigured your grid-mapfile or GUMS related configuration. If you have reasonable result you may try to srm copy.

If srm-ping is successful, you can test to copy local file to your BeStman-gateway-mode. First, create a file test1 in /tmp directory and execute:

 srm-copy   file:////tmp/test1 
srm://BeStMan_host:secure_port/srm/v2/server\?SFN=FS_ROOT_DIR/test1 -spacetoken %TOKEN_1_NAME

########################################### 
SRM_HOME is /usr/local/vdt_client/srm-client-lbnl 
JAVA_HOME is /usr/local/vdt_client/jdk1.5 
X509_CERT_DIR = /etc/grid-security/certificates 
GSI_DAEMON_TRUSTED_CA_DIR = /etc/grid-security/certificates 
########################################### 
SRM-CLIENT: Mon Nov 03 11:32:03 CST 2008 Connecting to 
httpg://fapl118.fnal.gov:8443/srm/v2/server 
 
SRM-CLIENT: Mon Nov 03 11:32:04 CST 2008 Calling SrmPrepareToPutRequest now ... request.token=put:5 status=SRM_SUCCESS explanation=null SRM-CLIENT: RequestFileStatus for SURL=file:////tmp/test1_1 is Ready. SRM-CLIENT: received TURL=gsiftp://fg0x5.fnal.gov//home/tlevshin/cache/test_4 >>>Total Memory=17932288 >>>Free Memory=6875256 >>>Memory in use=11057032 SRM-CLIENT: Mon Nov 03 11:32:08 CST 2008 start file transfer. SRM-CLIENT:Source=file:////tmp/test1_1 SRM-CLIENT:Target=gsiftp://fg0x5.fnal.gov//home/tlevshin/cache/test_4 SRM-CLIENT: Mon Nov 03 11:32:10 CST 2008 end file transfer. SRM-CLIENT: Mon Nov 03 11:32:10 CST 2008 Calling putDone for srm://fapl118.fnal.gov:8443/srm/v2/server?SFN=/home/tlevshin/cache/test_4 SRM-CLIENT: Mon Nov 03 11:32:18 CST 2008 end file transfer. SRM-CLIENT: Mon Nov 03 11:32:18 CST 2008 end file transfer. SRM-CLIENT: Request completed with success SRM-CLIENT: Printing text report now ... SRM-CLIENT*REQUESTTYPE=put SRM-CLIENT*TOTALFILES=1 SRM-CLIENT*TOTAL_SUCCESS=1 SRM-CLIENT*TOTAL_FAILED=0 SRM-CLIENT*REQUEST_TOKEN=put:5 SRM-CLIENT*REQUEST_STATUS=SRM_SUCCESS SRM-CLIENT*SOURCEURL[0]=file:////tmp/test1_1 SRM- CLIENT*TARGETURL[0]=srm://fapl118.fnal.gov:8443/srm/v2/server?SFN=/home/tlevshin/cache/test_4 SRM-CLIENT*TRANSFERURL[0]=gsiftp://fg0x5.fnal.gov//home/tlevshin/cache/test_4 SRM-CLIENT*ACTUALSIZE[0]=16 SRM-CLIENT*FILE_STATUS[0]=SRM_SUCCESS SRM-CLIENT*EXPLANATION[0]=SRM-CLIENT: PutDone is called successfully ExitCode=0

If you turned on Gratia GridFTP transfer probes, you should be able to see the accounting information by accessing your Gratia collector. See details in Preparing, Installing and Validating Gratia transfer probe.

Troubleshooting

File Locations

You could find log and configuration files for each of the module in the following location:

Module Name Configuration files Log files
BeStMan $VDT_LOCATION/bestman/conf/bestman.rc $VDT_LOCATION/vdt-app-data/bestman/logs/event.srm.log
$VDT_LOCATION/vdt-app-data/bestman/logs/bestman.log
GridFTP $VDT_LOCATION/vdt/services/vdt-run-gsiftp.sh.env $VDT_LOCATION/globus/var/log/gridftp.log
$VDT_LOCATION/globus/var/log/gridftp-auth.log

Open Ports

The following ports are opened for the installed services

Module Name Port Number Protocol
BeStMan default 10080 tcp
default 10443 tcp
GridFTP 2811 tcp
lowPort,maxPort if needed to control outbound globus connections tcp

Debugging Procedure

If system validation failed, you would probably need to check the each component in order to verify your installation. In order to do so you should check all of them in the following order:
  • GUMS (if in use)
  • GridFTP
  • BeStMan

Verifying GUMS

Make sure that the service certificate you specified for BeStMan configuration with --cert service_cert , --key service_key options and GridFTP service certificate are accepted by GUMS (see GUMS Installation Documentation)

Get mapping uid for your certificate and verify that this uid exists on BeStMan and GridFTP node.

Verifying GridFTP

Login on the node where you have installed have your certificate installed and access to http://software.grid.iu.edu/osg-1.2:wn_client or http://software.grid.iu.edu/osg-1.2:client.

You will need to get your voms-proxy or grid_proxy certificate see this section? :

Then test GridFTP:

source VDT_LOCATION/setup.sh 
echo “This is a test” >/tmp/test 
globus-url-copy -dbg file:///tmp/test gsiftp://GridFtp_host/tmp/test 

Check the GridFTP logs if you have encountered any errors.

Verifying BeStMan-gateway-mode

First, make sure that BeStMan is running:

# ps auxww|grep $VDT_LOCATION/bestman|grep -v grep
daemon   27648  0.0  0.0  4944 1168 pts/2    S    07:46   0:00 /bin/sh /usr/local/osg-bestman/bestman/sbin/bestman.server
daemon   27676  3.3  7.4 715240 155208 pts/2 Sl   07:46   1:59 /usr/local/osg-bestman/jdk1.6/bin/java -server -Xmx512m -Dorg.globus.tcp.port.range=20000,25000 -DX509_CERT_DIR=/usr/local/osg-bestman/globus/TRUSTED_CA -Daxis.ServerConfigFile=/usr/local/osg-bestman/bestman/conf/server-config.wsdd gov.lbl.srm.server.Server /usr/local/osg-bestman/bestman/conf/bestman.rc

If BeStMan is not running check information in that log files $VDT_LOCATION/vdt-install.log and $VDT_LOCATION/vdt-app-data/bestman/logs/bestman.log.

There is no error in the log file ($VDT_LOCATION/vdt-app-data/bestman/logs/event.srm.log)

References

Screen Dump of the Complete Install Process

Basic Installation and Configuration

Below is a screen dump of basic installation and configuration of BeStMan-Gateway:

  • using grid-mapfile
  • running on default ports
  • using default certificate and key files
  • not configuring storage area
  • not enabling gratia probe services
More... Close
# cd /usr/local
# cd pacman-3.29
# source setup.sh
# cd ..
# mkdir osg_1.2_bestman
# cd osg_1.2_bestman
# pacman -get http://software.grid.iu.edu/osg-1.2:Bestman
Do you want to add [http://software.grid.iu.edu/osg-1.2] to [trusted.caches]? (y/n/yall): yall
Beginning VDT prerequisite checking script vdt-common/vdt-prereq-check...       

All prerequisite checks are satisfied.
                                                                          


========== IMPORTANT ==========
Most of the software installed by the VDT *will not work* until you install
certificates.  To complete your CA certificate installation, see the notes
in the post-install/README file.

# source setup.sh

# $VDT_LOCATION/vdt/bin/vdt-ca-manage setupca --location local -url osg
Setting CA Certificates for VDT installation at '/usr/local/osg_1.2.6_bestman_gridmap'

Setup completed successfully.

# vdt-post-install
Starting...
Configuring PRIMA... Done.
Configuring EDG-Make-Gridmap... Done.
Completed all configuration.

# vdt-control --list
Service                 | Type   | Desired State
------------------------+--------+--------------
fetch-crl               | cron   | do not enable
vdt-rotate-logs         | cron   | do not enable
vdt-update-certs        | cron   | do not enable
gsiftp                  | inetd  | do not enable
gratia-gridftp-transfer | cron   | do not enable
gums-host-cron          | cron   | do not enable
edg-mkgridmap           | cron   | do not enable


# vdt-version

You have installed a subset of VDT version 2.0.0p13:

Software                                                 Status              
--------                                                 ------              
Berkeley Storage Manager (BeStMan) 2.2.1.3.8             OK                  
vdt-ca-manage 1.1                                        OK                  
vdt-update-certs 2.5                                     OK                  
CA Certificates 1.12 (includes IGTF 1.33 CAs)            -                   
EDG Make Gridmap 3.0.0                                   OK                  
Fetch CRL 2.6.6                                          OK                  
GPT 3.2-4.0.8p1                                          OK                  
Gratia GridFTP Probe 1.06.13b-1                          OK                  
Grid User Management System (GUMS) Client 1.3.17         OK                  
Java 5 SDK 1.5.0_21                                      OK                  
Java 6 SDK 1.6.0_16                                      OK                  
Logrotate 3.7                                            OK                  
PRIMA Authorization Module 0.8.4                         OK                  
pyOpenSSL module 0.9                                     OK                  
VOMS Client 1.8.8-2p1                                    OK                  
Wget 1.11.4                                              OK                  


Status legend:
OK: Software is up to date with the latest release in VDT version 2.0.0
- : Not enough information to determine if updates are available.
See man page for more information.

# osg-version
OSG 1.2.6

# $VDT_LOCATION/vdt/setup/configure_bestman --server y --enable-gateway
# vi /etc/sudoers 
.......
Cmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv, /bin/ls
Runas_Alias SRM_USR = ALL, !root
daemon ALL=(SRM_USR) NOPASSWD: SRM_CMD


# vdt-control -enable fetch-crl vdt-rotate-logs vdt-update-certs edg-mkgridmap
# edg/sbin/edg-mkgridmap 
# vdt-control -on

Login as user:

$ source /usr/local/osg_1.2.6_bestman/setup.sh
$ export PATH=/usr/local/osg_1.2.6_bestman/bestman/bin:$PATH
$ grid-proxy-init 
Your identity: /DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821
Enter GRID pass phrase for this identity:
Creating proxy ...................................... Done
Your proxy is valid until: Fri Feb 19 03:27:02 2010
$ srm-ping srm://fg0x5.fnal.gov:10443
srm-ping   2.2.1.3.8   Wed Dec  2 22:54:35 PST 2009
SRM-Clients and BeStMan Copyright(c) 2007-2009,
Lawrence Berkeley National Laboratory. All rights reserved.
Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman
 

SRM-CLIENT: SURL does not contains ?SFN 
SRM-CLIENT: serviceHandle /srm/v2/server is taken from the srmclient.conf 
SRM-CLIENT: SFN is assumed as 
SRM-CLIENT: Connecting to serviceurl httpg://fg0x5.fnal.gov:10443/srm/v2/server

SRM-PING: Thu Feb 18 15:27:51 CST 2010  Calling SrmPing Request...
versionInfo=v2.2

Extra information (Key=Value)
backend_type=BeStMan
backend_version=2.2.1.3.8
backend_build_date=2009-12-03T05:09:16.000Z 
gsiftpTxfServers[0]=gsiftp://fg0x5.fnal.gov
GatewayMode=Enabled
clientDN=/DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821
localIDMapped=tlevshin

$ echo "This is a test " > /tmp/test

$ srm-copy file:///tmp/test  srm://fg0x5.fnal.gov:10443/srm/v2/server\?SFN=/tmp/test_tanya
srm-copy   2.2.1.3.8   Wed Dec  2 22:54:35 PST 2009
SRM-Clients and BeStMan Copyright(c) 2007-2009,
Lawrence Berkeley National Laboratory. All rights reserved.
Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman
 
SRM-CLIENT: Thu Feb 18 15:30:42 CST 2010 Connecting to httpg://fg0x5.fnal.gov:10443/srm/v2/server

SRM-CLIENT: Thu Feb 18 15:30:43 CST 2010 Calling SrmPrepareToPutRequest now ...
request.token=put:0
Request.status=SRM_SUCCESS
explanation=null
SRM-CLIENT: received TURL=gsiftp://fg0x5.fnal.gov//tmp/test_tanya

SRM-CLIENT: Thu Feb 18 15:30:45 CST 2010 start file transfer
SRM-CLIENT:Source=file:////tmp/test
SRM-CLIENT:Target=gsiftp://fg0x5.fnal.gov//tmp/test_tanya

SRM-CLIENT: Thu Feb 18 15:30:47 CST 2010 end file transfer for file:///tmp/test

SRM-CLIENT: Thu Feb 18 15:30:47 CST 2010 Calling putDone for srm://fg0x5.fnal.gov:10443/srm/v2/server?SFN=/tmp/test_tanya
Result.status=SRM_SUCCESS
Result.Explanation=null

SRM-CLIENT: Request completed with success

SRM-CLIENT: Printing text report now ...

SRM-CLIENT*REQUESTTYPE=put
SRM-CLIENT*TOTALFILES=1
SRM-CLIENT*TOTAL_SUCCESS=1
SRM-CLIENT*TOTAL_FAILED=0
SRM-CLIENT*REQUEST_TOKEN=put:0
SRM-CLIENT*REQUEST_STATUS=SRM_SUCCESS
SRM-CLIENT*SOURCEURL[0]=file:///tmp/test
SRM-CLIENT*TARGETURL[0]=srm://fg0x5.fnal.gov:10443/srm/v2/server?SFN=/tmp/test_tanya
SRM-CLIENT*TRANSFERURL[0]=gsiftp://fg0x5.fnal.gov//tmp/test_tanya
SRM-CLIENT*ACTUALSIZE[0]=15
SRM-CLIENT*FILE_STATUS[0]=SRM_SUCCESS
SRM-CLIENT*EXPLANATION[0]=SRM-CLIENT: PutDone is called successfully

Advanced Installation and Configuration

The is a screen dump of basic installation and configuration of BeStMan:

  • using GUMS for authorization
  • running on specified ports
  • using different certificate and key files
  • specifying a particular storage area
  • using space tokens
  • enabling gratia probe services
More... Close
# cd /usr/local
# mkdir osg_1.2.6_bestman
# cd osg_1.2.6_bestman
# export VDT_GUMS_HOST=gums.fnal.gov
# source ../pacman-3.29/setup.sh
# pacman -get http://software.grid.iu.edu/osg-1.2:Bestman
Do you want to add [http://software.grid.iu.edu/osg-1.2] to [trusted.caches]? (y/n/yall): yall
Beginning VDT prerequisite checking script vdt-common/vdt-prereq-check...       

All prerequisite checks are satisfied.
                                                                          


========== IMPORTANT ==========
Most of the software installed by the VDT *will not work* until you install
certificates.  To complete your CA certificate installation, see the notes
in the post-install/README file.



# source setup.sh

#  $VDT_LOCATION/vdt/bin/vdt-ca-manage setupca --location local -url osg
Setting CA Certificates for VDT installation at '/usr/local/osg_1.2.6_bestman'

Setup completed successfully.
# vdt-post-install
Starting...
Configuring PRIMA... Done.
Configuring EDG-Make-Gridmap... Done.
Completed all configuration.
# $VDT_LOCATION/vdt/setup/configure_bestman --server y --cert /etc/grid-security/bestman_cert/bestmancert.pem --key /etc/grid-security/bestman_cert/bestmankey.pem --gums-host gums.fnal.gov --gums-port 8443 --with-transfer-servers gsiftp://fg0x5.fnal.gov --with-tokens-list "FermiDATADISK[desc:FERMIDATADISK][9] [retention:CUSTODIAL][latency:ONLINE][usedBytesCommand:/usr/bin/du -s -b /cache]" --http-port 10080 --https-port 10443 --user daemon --with-allowed-paths /cache --enable-gateway


#vi /etc/sudoers 
.......
Cmnd_Alias SRM_CMD = /bin/rm, /bin/mkdir, /bin/rmdir, /bin/mv, /bin/ls
Runas_Alias SRM_USR = ALL, !root
daemon ALL=(SRM_USR) NOPASSWD: SRM_CMD


# cp post-install/prima-authz.conf /etc/grid-security/
# cp post-install/gsi-authz.conf  /etc/grid-security/

# vdt-control --list
Service                 | Type   | Desired State
------------------------+--------+--------------
fetch-crl               | cron   | do not enable
vdt-rotate-logs         | cron   | do not enable
vdt-update-certs        | cron   | do not enable
gsiftp                  | inetd  | do not enable
gratia-gridftp-transfer | cron   | do not enable
gums-host-cron          | cron   | do not enable
edg-mkgridmap           | cron   | do not enable
bestman                 | init   | enable

# $VDT_LOCATION/vdt/setup/configure_gratia --probe-cron --force-probe-config --site-name FERMI_TEST_1 --report-to gratia-osg-itb.opensciencegrid.org:8881 --probe gridftp-transfer

# vdt-control --enable fetch-crl  vdt-rotate-logs vdt-update-certs  gums-host-cron vdt-update-certs gsiftp gratia-gridftp-transfer
running 'vdt-register-service --name fetch-crl --enable'... ok
running 'vdt-register-service --name vdt-update-certs --enable'... ok
running 'vdt-register-service --name gums-host-cron --enable'... ok
running 'vdt-register-service --name vdt-update-certs --enable'... ok
running 'vdt-register-service --name gsiftp --enable'... ok
running 'vdt-register-service --name gratia-gridftp-transfer --enable'... ok

# vdt-control -on
enabling cron service fetch-crl... ok
skipping cron service 'vdt-rotate-logs' -- marked as disabled
enabling cron service vdt-update-certs... ok
enabling inetd service gsiftp... ok
enabling cron service gratia-gridftp-transfer... ok
enabling cron service gums-host-cron... ok
skipping cron service 'edg-mkgridmap' -- marked as disabled
enabling init service bestman... ok


#mkdir /cache
#chmod 777 /cache
Login as user:
$ source /usr/local/osg_1.2.6_bestman/setup.sh
$ export PATH=/usr/local/osg_1.2.6_bestman/bestman/bin:$PATH
$ grid-proxy-init 
Your identity: /DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821
Enter GRID pass phrase for this identity:
Creating proxy ...................................... Done
Your proxy is valid until: Fri Feb 19 03:27:02 2010


$  export PATH=/usr/local/osg_1.2.6_bestman/bestman/bin:$PATH
-bash-3.00$  srm-ping srm://fg0x5.fnal.gov:10443
srm-ping   2.2.1.3.8   Wed Dec  2 22:54:35 PST 2009
SRM-Clients and BeStMan Copyright(c) 2007-2009,
Lawrence Berkeley National Laboratory. All rights reserved.
Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman
 

SRM-CLIENT: SURL does not contains ?SFN 
SRM-CLIENT: serviceHandle /srm/v2/server is taken from the srmclient.conf 
SRM-CLIENT: SFN is assumed as 
SRM-CLIENT: Connecting to serviceurl httpg://fg0x5.fnal.gov:10443/srm/v2/server

SRM-PING: Thu Feb 18 16:03:46 CST 2010  Calling SrmPing Request...
versionInfo=v2.2

Extra information (Key=Value)
backend_type=BeStMan
backend_version=2.2.1.3.8
backend_build_date=2009-12-03T05:09:16.000Z 
gsiftpTxfServers[0]=gsiftp://fg0x5.fnal.gov
GatewayMode=Enabled
clientDN=/DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821
gumsIDMapped=fnalgrid
staticToken(0)=FermiDATADISK desc=FERMIDATADISK size=9663676416

$ srm-copy file:///tmp/test  srm://fg0x5.fnal.gov:10443/srm/v2/server\?SFN=/cache/fnalgrid/test_tanya  -mkdir
srm-copy   2.2.1.3.8   Wed Dec  2 22:54:35 PST 2009
SRM-Clients and BeStMan Copyright(c) 2007-2009,
Lawrence Berkeley National Laboratory. All rights reserved.
Support at SRM@LBL.GOV and documents at http://datagrid.lbl.gov/bestman
 
SRM-CLIENT: Thu Feb 18 16:23:08 CST 2010 Connecting to httpg://fg0x5.fnal.gov:10443/srm/v2/server

SRM-CLIENT: Thu Feb 18 16:23:08 CST 2010 Calling SrmPrepareToPutRequest now ...
request.token=put:2
Request.status=SRM_SUCCESS
explanation=null
SRM-CLIENT: received TURL=gsiftp://fg0x5.fnal.gov//cache/fnalgrid/test_tanya

SRM-CLIENT: Thu Feb 18 16:23:10 CST 2010 start file transfer
SRM-CLIENT:Source=file:////tmp/test
SRM-CLIENT:Target=gsiftp://fg0x5.fnal.gov//cache/fnalgrid/test_tanya

SRM-CLIENT: Thu Feb 18 16:23:13 CST 2010 end file transfer for file:///tmp/test

SRM-CLIENT: Thu Feb 18 16:23:13 CST 2010 Calling putDone for srm://fg0x5.fnal.gov:10443/srm/v2/server?SFN=/cache/fnalgrid/test_tanya
Result.status=SRM_SUCCESS
Result.Explanation=null

SRM-CLIENT: Request completed with success

SRM-CLIENT: Printing text report now ...

SRM-CLIENT*REQUESTTYPE=put
SRM-CLIENT*TOTALFILES=1
SRM-CLIENT*TOTAL_SUCCESS=1
SRM-CLIENT*TOTAL_FAILED=0
SRM-CLIENT*REQUEST_TOKEN=put:2
SRM-CLIENT*REQUEST_STATUS=SRM_SUCCESS
SRM-CLIENT*SOURCEURL[0]=file:///tmp/test
SRM-CLIENT*TARGETURL[0]=srm://fg0x5.fnal.gov:10443/srm/v2/server?SFN=/cache/fnalgrid/test_tanya
SRM-CLIENT*TRANSFERURL[0]=gsiftp://fg0x5.fnal.gov//cache/fnalgrid/test_tanya
SRM-CLIENT*ACTUALSIZE[0]=15
SRM-CLIENT*FILE_STATUS[0]=SRM_SUCCESS
SRM-CLIENT*EXPLANATION[0]=SRM-CLIENT: PutDone is called successfully

Comments

PM2RPM?_TASK = SE RobertEngel 28 Aug 2011 - 05:49

Topic attachments
I Attachment Action Size Date Who Comment
jpgjpg bestman-gateway-howitworks.jpg manage 49.7 K 19 Mar 2009 - 19:37 AlexSim BeStMan-Gateway - How it works
jpgjpeg bestman_gateway_arch.jpeg manage 9.7 K 18 Feb 2009 - 22:37 TanyaLevshina BeStMan-gateway architecture
Topic revision: r100 - 15 Feb 2012 - 21:00:00 - KyleGross
Hello, TWikiGuest
Register

Introduction

Installation and Update Tools

Clients

Compute Element

Storage Element

Other Site Services

VO Management

Software and Caches

Central OSG Services

Additional Information

Community
linkedin-favicon_v3.icoLinkedIn
FaceBook_32x32.png Facebook
campfire-logo.jpgChat
 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..