Documentation/Release3
GipConfiguration
Owner AnthonyTiradani
Area ComputeElement
Role SysAdmin
Type Installation
Reviewer BrianBockelman Owner AnthonyTiradani
In Progress Not Released

Configuration of the Generic Information Provider

About this Document

hand This document is for System Administrators. It contains the procedure to configure the Generic Information Provider.

Configuration of the Generic Information Provider (GIP)

One important aspect of a site being part of the Open Science Grid is the ability of the Compute Element to describe the site to external users. The Generic Information Provider (GIP) generates site information and provides it in the GLUE schema.

The GIP is collecting information about some aspects of your site including its hardware composition, the batch system and associated storage. It relies on the configuration files located in /etc/osg/config.d to collect the information.

The following sections detail information about the options in the [GIP], [Subcluster], and [SE] sections of the configuration files.

For Sites with Multiple Compute Elements

If you would like to properly advertise multiple Compute Elements per resource, make sure to:

  • set the value of cluster_name in the [GIP] section to be the same for each CE
  • set the value of other_ces in the [GIP] section to be the hostname of the other CEs at your site; this should be comma separated. So, if you have two CEs, ce1.example.com and ce2.example.com, the value of other_ces on ce1.example.com should be "ce2.example.com. This assumes that the same queues are visible on each CE.
  • set the value of resource_group in the "Site Information" section to be the same for each CE.
  • the value of resource in the "Site Information" section should be CE unique.
  • Have the exact same configuration values for the GIP, SE*, and Subcluster* sections in each CE.

It is good practice to run "diff" between the config files of the different CEs. The only changes should be the value of localhost in the [DEFAULT] section and the value of other_ces in the [GIP] section.

Subcluster Configuration

Another aspect of the GIP is to advertise the physical hardware a job will encounter when submitted to your site. This is information is provided to GIP in the [Subcluster <name>] section of the resource configuration file /etc/osg/config.d/30-gip.ini.

A sub-cluster is defined to be a homogeneous set of worker node hardware. At least one Subcluster section is required. For WLCG sites, information filled in here will be advertised as part of your MoU commitment, so please strive to make sure it is correct.

For each sub-cluster constituting your site, fill in the information about the worker node hardware by creating a new section choosing a unique name using the following format: [Subcluster <name>] where <name> is the sub-cluster name.

ALERT! IMPORTANT
This sub-cluster name must be unique for the entire OSG. Do not pick something generic like [Subcluster Opterons]!

Examples can be found here. This page shows the subcluster mapping from attribute names in /etc/osg/config.d/30-gip.ini to GLUE attribute names.

The values for this section are relatively well-documented and self-explanatory and are given below:

Option Valid Values Explanation
name String This is the same name that is in the Section label. It should be globally unique!
node_count Positive Integer This is the number of worker nodes in the sub-cluster
ram_mb Positive Integer Megabytes of RAM per node.
cpu_model String CPU model, as taken from /proc/cpuinfo. Please, no abbreviations!
cpu_vendor String Vendor's name: AMD, Intel, or any other.
cpu_speed_mhz Positive Integer Approximate speed in MHZ of the CPU as taken from /proc/cpuinfo
cpus_per_node Positive Integer Number of CPUs (physical chips) per node.
cores_per_node Positive Integer Number of cores per node.
inbound_network True or False Set to true or false depending on inbound connectivity. That is, external hosts can contact the worker nodes in this sub-cluster based on their hostname.
outbound_network True or False Set to true or false depending on outbound connectivity. Set to true if the worker nodes in this sub-cluster can communicate with the internet.
cpu_platform x86_64 or i686 NEW for OSG 1.2. Set according to your sub-cluster's processor architecture.
HEPSPEC Positive Integer Optional: Publish the HEPSPEC number for the subcluster.
SI00 Positive Integer Optional: Publish the SpecInt2000? number for the subcluster. Default: 2000
SF00 Positive Integer Optional: Publish the SpecFloat2000? number for the subcluster. Default: 2000

For Compute Elements

The GIP queries the batch system specified and enabled in /etc/osg/config.d/20-.ini. Alternatively you may manually specify which batch system to query by setting the batch attribute in the [GIP] section.

Information on Condor

GIP has several options that will control how it interacts with the Condor Batch system. You can determine which Condor daemon GIP queries as well as set whitelists and black lists that GIP will use to properly report which VOs are assigned to which Condor Groups.

Option Valid Values Explanation
use_collector True or False Ordinarily GIP queries the Negotiator. This option allows the administrator to tell GIP to query the collector instead of the negotiator. This option is overridden by query_only_local_condor
query_only_local_condor True or False This option overrides use_collector. It causes GIP to query the local Condor SCHEDD rather than the Negotiator or Collector. If not explicitly set, this option defaults to False
exclude_schedds String This is a comma separated list of schedds that should be excluded during a condor_status -submitter query
subtract_owner True or False By default, owner VMs are not accounted for the total CPU numbers published by GIP since according to Condor they aren't available for user jobs. Set to False to tell GIP to not subtract CPUs provided by owner VMs
(groupname)_blacklist String Either set to "*" which denies access to all VOs to the Condor group identified by (groupname) or set to a comma separate list of VOs you wish to deny access.
(groupname)_whitelist String Either set to "*" which allows access to all VOs to the Condor group identified by (groupname) or set to a comma separate list of VOs you wish to allow access.
jobs_constraint String adds a --constraint option to the condor_q command that GIP will execute. %NOTE: for big sites this could cause rather large performance issues
status_constraint String adds a --constraint option to the condor_status command that GIP will execute. %NOTE: for big sites this could cause rather large performance issues
max_wall Integer This allows an administrator to publish the maximum wall time allowed on the Condor pool

HELP NOTE
To generate a list of supported VOs for a queue, the blacklist is evaluated before the whitelist.

Information on PBS

By default any authorized grid user may submit to every PBS queue listed in /usr/share/globus/globus_gram_job_manager/pbs.rvf. In this case GIP automatically advertises all listed queues.

GIP will also detect if queues restrict access to certain users (acl_users) or certain groups (acl_groups). A reverse mapping from these users and groups to their associated Virtual Organization provides access information of VOs to certain queues which will be advertised by the GIP instead.

This process is not perfect and sometimes fails to generate the right information. In this case you may manually blacklist and whitelist PBS queues for listed VOs in the [PBS] section of the resource configuration file /etc/osg/config.d/20-pbs.ini.

GIP also allows you to completely exclude queues from the list of queues that GIP will advertise.

Option Valid Values Explanation
host String Optional - This is the hostname that will be appended to PBS commands if set. Example: qstat -B -f host
preemption Integer Optional - Set to 1 in order enable reporting that preemption is enabled on the cluster. Default is 0
pbs_path String Optional - This allows an administrator to set a non standard PBS location or a PBS location that is not in the system search path
(queuename)_blacklist String Either set to "*" which denies access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to deny access.
(queuename)_whitelist String Either set to "*" which allows access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to allow access.
queue_exclude String Comma-separated list of queue names that GIP should exclude from the list of queues to publish

HELP NOTE
To generate a list of supported VOs for a queue, the blacklist is evaluated before the whitelist.

Information on SGE

The home attribute in the [SGE] section of the resource configuration file /etc/osg/config.d/20-sge.ini has been replaced by sge_root and sge_cell, which should be set to the value of of $SGE_ROOT and $SGE_CELL, respectively. The GIP assumes that it can source $SGE_ROOT/$SGE_CELL/common/settings.sh to create a working SGE environment.

Option Valid Values Explanation
preemption Integer Optional - Set to 1 in order enable reporting that preemption is enabled on the cluster. Default is 0
sge_path String Optional - This allows an administrator to set a non standard SGE location or a SGE location that is not in the system search path
sge_root String This value should be set to $SGE_ROOT - GIP assumes that it can source $SGE_ROOT/$SGE_CELL/common/settings.sh to create a working SGE environment.
sge_cell String This value should be set to $SGE_CELL - GIP assumes that it can source $SGE_ROOT/$SGE_CELL/common/settings.sh to create a working SGE environment.
(queuename)_blacklist String Either set to "*" which denies access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to deny access.
(queuename)_whitelist String Either set to "*" which allows access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to allow access.
queue_exclude String Comma-separated list of queue names that GIP should exclude from the list of queues to publish

Information on LSF

Option Valid Values Explanation
host String Optional - This is the hostname that will be appended to LSF commands if set. Example: lshosts -w host
preemption Integer Optional - Set to 1 in order enable reporting that preemption is enabled on the cluster. Default is 0
lsf_location String Optional - This allows an administrator to set a non standard LSF location or a LSF location that is not in the system search path
lsf_profile String Optional - Set this to specify the location of the LSF profile file. GIP will source this to extract a working LSF environment. Defaults to /lsf/conf/profile.lsf
(queuename)_blacklist String Either set to "*" which denies access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to deny access.
(queuename)_whitelist String Either set to "*" which allows access to all VOs to the queue identified by (queuename) or set to a comma separate list of VOs you wish to allow access.
queue_exclude String Comma-separated list of queue names that GIP should exclude from the list of queues to publish

Link the Compute Element to an external Storage Element

A compute element may provide mass storage to grid users on an external storage element. To advertise the available storage space on the external storage element create the file /etc/gip/gip.conf with following content:

[cesebind]
; Advertise the availability of an external storage element for mass storage use on the compute element.
simple = False
se_list = <coma-separated list of the SRM endpoint of storage elements>

For Storage Elements

For each Storage Element add a new section to the resource configuration file /etc/osg/config.d/30-gip.ini named [SE_<name>] where <name> is the unique name of your SE. The following sections illustrate the configuration process for a:

  1. Generic Storage Element
  2. BeStMan Storage Element
  3. dCache Storage Element

See following examples for working SE configuration sections.

HELP NOTE
WLCG sites must use the bestman or dcache19 provider in order to be advertised correctly.

Generic Storage Element

Following attributes are available to configure the information provided by GIP for a generic storage element:

Option Valid Values Explanation
enabled True or False Indicates whether or not this SE section is enabled for the GIP.
name String Name of the Storage Element as registered in OSG Information Management System.
srm_endpoint String The endpoint of the SE. It MUST contain the hostname, port, and the server location (such as /srm/v2/server). It MUST NOT have the ?SFN= string. Example: srm://srm.example.com:8443/srm/v2/server
srm_port String Sets the specific port number that will appear in the srm_endpoint.
provider_implementation String Set to static for a generic SE.
Set to bestman for BeStMan SE.
Set to dcache or dcache19 for dCache SE.
implementation String Name of the SE implementation.
version String Version number of the SE implementation.
default_path String Default storage path. For the srm command line this corresponds to the path that appears after the question mark. If a path has the string VONAME in it, that string will be replaced with the appropriate vo name.
use_df True or False Set to True if the SE provider may use df on the directory referenced in default_path to obtain the size of the SE.
vo_dirs Comma-separated List A comma-separated list of <VONAME>:<PATH> pairs. The PATH will override the default_path attribute for VONAME.
allowed_vos Comma-separated List By default all VOs are advertised to have access to the SE. To advertise only a subset of VOs, provide a comma-separated list here.
mount_point String Used if your SE provides a POSIX-like mount (e.g. Lustre, HDFS, xrootdfs). The value of `mount_point` should be two paths; first, the path where the file system is mounted on the worker nodes, followed by the exported directory of the file system.

BeStMan Storage Element

For BeStMan storage elements, there are a few notes to consider for the SE attributes:

Option Valid Values Explanation
srm_endpoint String The endpoint of the SE. It is most likely of the form srm://srm.example.com:8443/srm/v2/server.
provider_implementation String Set to bestman.
implementation String Set to bestman.

The BeStMan provider may use srm-ping to query a BeStMan storage element for information. This means that the Unix tomcat user will need to access to a valid http service certificate in /etc/grid-security/http/httpcert.pem and its associated key. Instructions on how to request and install a service certificate can be found here.

dCache Storage Element

Option Valid Values Explanation
srm_endpoint String The endpoint of the SE. It is most likely of the form srm://srm.example.com:8443/srm/managerv2.
provider_implementation String Set to dcache19 for dCache 1.9 sites.
Set to static for default values.
If you use the dcache provider with dCache 1.8, see this page to complete installation. If you use the dcache19 provider, you must fill in the location of your dCache's information provider
infoprovider_endpoint String Url to the dcache information provider. Only required for the dcache19 provider.
implementation String Set to dcache.
version String The dCache version.
Will be auto-detected in the case of non-static providers.

The implementation of dCache version 1.9 added a new read-only XML information service which feeds the dcache19 provider. The location of the info provider is most likely http://dcache_head_node.example.com:2288/info. It runs on the same node that the dCache web interface. Also consult your dCache documentation on how to enable the information service.

Advertise Available Storage Space

The GIP has a concept of space as the logical grouping of available storage space. The static provider just advertises the storage element as one homogeneous storage space. The dCache and BeStMan can also partition the storage space into several sub-spaces.

  • For dCache a space may be either a link group or a pool group. If a link group contains one or more pool groups, only the link group will be advertised. The space name is the name of the link group or pool group.

  • For BeStMan a space is a SRM space token. The name of the space is the description of the space token.

Both dCache and BeStMan storage elements will try to detect the correct values automatically. To manually change attributes in the SE section of the resource configuration file /etc/osg/config.d/30-gip.ini consult the table of available space-related attributes:

Option Valid Values Explanation
spaces Comma-separated List List of all the spaces that should be restricted by VO.
space_<NAME>_vos Comma-separated List By default, all VOs are allowed to access all spaces. To restrict access by VO provide a list of VOs that may access the storage space <NAME>.
space_<NAME>_default_path String The default storage path for VOs for the space named <NAME>. As in the "default_path" option, the word VONAME is evaluated to be the VO's name.
space_<NAME>_path Comma-separated List A list of <VONAME>:<PATH> pairs for this space.

Advertise Available Services

The [GIP] section of the resource configuration file /etc/osg/config.d/30-gip.ini provides the possibility to advertise services available at your site.

However, Multi-CE sites MUST edit both cluster_name and other_ces.

All options are given in the table below:

Option Valid Values Explanation
advertise_gums True or False Defaults to False. If you want GIP to query and advertise your gums server set this to True.
advertise_gsiftp True or False Defaults to True. If you don't want GIP to advertise your gridftp server set this to False.
gsiftp_host String This should be set to the name of the gridftp server GIP will advertise if the advertise_gridftp setting is set to True.
cluster_name String This should only be set if you run multiple gatekeepers for the same cluster; if you do, set this value to the FQDN of the head node of the cluster.
other_ces String This should only be set if you run multiple gatekeepers for the same cluster; if you do, set this value to the comma-separate list of FQDNs for the other CEs at this site.
local_template_dirs String Support for local template dirs (contributed by Sam Morrison @ ARCS)

If you want to advertise glexec support you must set glexec_location in the [Misc] section. The value should be the location of glexec on the worker node.

Advertise WLCG Attributes

Sites that are interoperable with the WLCG will need to publish certain extra attributes that help identify them in the information systems. These options go in the [site] section in /etc/osg/config.d/40-siteinfo.ini. Some of this information is available at the CERN gridops site.
Option Valid Values Explanation
wlcg_tier Positive_Integer The WLCG computing model is composed of tiers. Put the Tier number that your site belongs to here.
wlcg_parent String Put your wlcg_tier - 1 site name that your site is assigned to here. For example, if your site is a Tier-2 site, then put the site name of the Tier-1 site that you are assigned to.
wlcg_name String String giving the name that the WLCG uses to identify the site
wlcg_grid String String defining which native grid the site belongs to. If not set, it will default to OSG. Example: OSG sites would set this value to OSG

Configuring GIP for SE Only

NOTE: The following instructions are preserved for OSG 1.2.x installations. To configure an SO only setup for GIP with OSG 3.x, please see Configure GIP on SE Only install

Now, you have to configure GIP. Several manual steps are required to do so:

  1. Create gip.conf file.
    # cd VDT_LOCATION/gip/etc
    # cp gip.se_only.example.conf gip.conf 
    
  2. Edit gip.conf file and modify this content:
    [gip]
    override = False
    check_osg = False
    osg_config = VDT_LOCATION/gip/etc/config.ini
    
    [vo]
    user_vo_map = VDT_LOCATION/gip/etc/osg-user-vo-map.txt
    
    the following way:
    [gip]
    override = False
    check_osg = False
    osg_config = VDT_LOCATION/gip/etc/config.ini
    
    [vo]
    user_vo_map = VDT_LOCATION/monitoring/osg-user-vo-map.txt
    
  3. Fill out a config.ini for the SE only site. The known minimum sections are: DEFAULT, Site Information, Storage, GIP, and SE <name>.
  • the Storage section only needs:
       se_available = %(enable)s
       default_se = <se fqdn>
    
  • the GIP section only needs:
       advertise_gsiftp = %(disable)s
    In order to modify the file you will need first to copy config.ini.se_only.example to config.ini first.
    # cp config.ini.se_only.example config.ini
    # vi config.ini
    
    You will need to modify config.ini accordingly following the instruction given above. Here are two examples for
  • dCache installation: More... Close
    [DEFAULT]
    unavailable = UNAVAILABLE
    default = UNAVAILABLE
    disable = False
    enable = True
    localhost = fg0x5.fnal.gov
    admin_email = tlevshin@fnal.gov
    osg_location = /opt/vdt_cemon
    
    [Site Information]
    resource =  FNAL-ITB-GIP-TEST1
    resource_group =  FNAL-ITB-GIP-TEST1
    sponsor = cms:100
    site_policy = http://www.uscms.org/SoftwareComputing/Grid/Policy/OSG_VO.html
    contact = cms-t1@fnal.gov
    city = Batavia, IL
    country = USA
    longitude = -88.2546
    latitude = 41.8412
    group = OSG-ITB
    email = %(admin_email)s
    host_name = %(localhost)s
    
    [Storage]
    se_available = %(enable)s
    default_se = cmssrm.fnal.gov
    
    [GIP]
    advertise_gsiftp = %(disable)s
    
    [SE dCache]
    enabled = True
    name = cmssrm.fnal.gov
    srm_endpoint = httpg://gwdca04.fnal.gov:8443/srm/managerv2
    provider_implementation = dcache19
    implementation = dcache
    version = 1.9.5-19
    default_path = /
    infoprovider_endpoint = http://gwdca06.fnal.gov/site/info
    
  • BeStMan installation: More... Close
    [DEFAULT]
    unavailable = UNAVAILABLE
    default = UNAVAILABLE
    disable = False
    enable = True
    localhost = fg0x5.fnal.gov
    admin_email = tlevshin@fnal.gov
    osg_location = /opt/vdt_cemon
    
    [Site Information]
    resource = FNAL-ITB-GIP-TEST1
    resource_group = FNAL-ITB-GIP-TEST1
    sponsor = cms:100
    site_policy = http://www.uscms.org/SoftwareComputing/Grid/Policy/OSG_VO.html
    contact = tlevshin@fnal.gov
    city = Batavia, IL
    country = USA
    longitude = -88.2546
    latitude = 41.8412
    group = OSG-ITB
    email = %(admin_email)s
    host_name = %(localhost)s
    
    [Storage]
    se_available = %(enable)s
    default_se = fg0x5.fnal.gov
    
    [GIP]
    advertise_gsiftp = %(disable)s
    advertise_accesspoints = False
    
    [SE BeStMan]
    enabled = True
    name = fg0x5.fnal.gov
    srm_endpoint = httpg://fg0x5.fnal.gov:10443/srm/v2/server
    provider_implementation = BeStMan
    implementation = BeStMan
    version = 2.2.2.0.0
    default_path = /cache
    

Next step is to prepare run_gip.sh script:

# cd gip/bin
# cp run_gip.sh.se_only.example run_gip.sh
# vi run_gip.sh
modify run_gip.sh accordingly:
export VDT_LOCATION=VDT_LOCATION
export GIP_LOCATION=VDT_LOCATION/gip
$GIP_LOCATION/bin/gip_info

You will need to set right permission on $VDT_LOCATION/gip/var directory. It should belong to the user that is running tomcat service (daemon).

# cd $VDT_LOCATION/gip
# chown daemon.daemon var

Comments

https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/GipConfiguration
PM2RPM?_TASK = CE RobertEngel 28 Aug 2011 - 06:06

-- AnthonyTiradani - 10 Jan 2012

Topic revision: r9 - 28 Jun 2012 - 16:18:51 - AnthonyTiradani
Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..