Ini Configuration Options

About this Document

hand This document outlines the settings and options found in the ini files for system administers that are installing and configuring osg software.

This page gives an overview of the options for each of the sections of the configuration files that osg-configure uses.

Conventions

In the tables below:

  • Mandatory options for a section are given in bold type. Sometime the default value may be OK and no edit required, but the variable has to be in the file.
  • Options that are not found in the default ini file are in italics.

Introduction

osg-configure and the INI files in /etc/osg/config.d allow a higl level configuration of OSG services.

Layout

The configuration files used by osg-configure are the one supported by Python's SafeConfigParser, similar in format to the INI configuration file used by MS Windows. Long lines can be split up using continuations as specified in email RFC 822 (each White Space Character can be preceded by a Newline to fold/continue the fileld on a new line). A config file is separated into sections with each section starting with a section name in square brackets (e.g. [Section 1]) . Lines that begin with a ; or a # are treated as comments and ignored. Options can be set using either name : value or name=value pairs. Finally, variables and variable substitutions are supported.

osg-configure reads and uses all of the files in /etc/osg/config.d that have a "ini" suffix. The files in this directory are ordered with a numeric prefix with higher numbers being applied later and thus having higher precedence (e.g. 00-foo.ini has a lower precedence than 99-local-site-settings.ini). Configuration sections and options can be specified multiple times in different files. E.g. a section called [PBS] can be given in 20-pbs.ini as well as 99-local-site-settings.ini.

Each of the files are successively read and merged to create a final configuration that is then used to configure OSG software. Options and settings in files read later override the ones in previous files. This allows admins to create a file with local settings (e.g. 99-local-site-settings.ini) that can be read last and which will be take precedence over the default settings in configuration files installed by various RPMs and which will not be overwritten if RPMs are updated. Note: This works well for osg-configure, but may complicate the GIP (information gathering), because GIP information in all files will be included. If you do this, you should remove the contents of 30-gip.ini, but not the file itself.

Variables and variable substitution

The osg-configure parser allows variables to be defined and used in the configuration file. Any option set in a given section can be used as a variable in that section. Note, this means that you will need to be careful when naming variables in order to avoid an infinite loop when resolving the variable substitution. Assuming that you have set an option with the name myoption in the section, you can substitute the value of that option elsewhere in the section by referring to it as %(myoption)s . Please note that the trailing s is required. Also the variable substitution will only occur when setting an option, so option names can not have a variable substitution in them.

Options and settings

If a setting is set to UNAVAILABLE or DEFAULT or left blank, osg-configure will try to use a sensible default for setting if possible.

Ignore setting

The enabled option, specifying whether a service is enabled or not, is a boolean but also accepts Ignore as a possible value. Using Ignore, results in the service associated with the section being ignored entirely (and any configuration is skipped). This differs from using False (or the %(disabled)s variable), because using False results in the service associated with the section being disabled. osg-configure will not change the configuration of the service if the enabled is set to Ignore.

This is useful, if you have a complex configuration for a given that can't be set up using the ini configuration files. You can manually configure that service by hand editing config files, manually start/stop the service and then use the Ignore setting so that osg-configure does not alter the service's configuration and status.

Configuration sections

The OSG configuration is divided into sections with each section starting with a section name in square brackets (e.g. [Section 1]). The configuration is split in multiple files and options form one section can be in more than one files.

The following sections give an overview of the options for each of the sections of the configuration files that osg-configure uses.

Site Information

The settings found in the Site Information section are described below. This section is used to give information about a resource such as resource name, site sponsors, administrators, etc. In the default installation, this section is found in /etc/osg/config.d/40-siteinfo.ini .

Option Values Accepted Description
group OSG , OSG-ITB This should be set to either OSG or OSG-ITB depending on whether your resource is in the OSG or OSG-ITB group. Most sites should specify OSG
host_name String This should be set to be hostname of the CE that is being configured
resource String The resource name of this CE endpoint as registered in OIM.
resource_group String The resource_group of this CE as registered in OIM.
sponsor String This should be set to the sponsor of the resource, if your resource has multiple sponsors, you can separate them using commas or specify the percentage using the following format 'osg, atlas, cms' or 'osg:10, atlas:45, cms:45'. The percentages must add up to 100 if multiple sponsors are used. If you have a sponsor that is not an OSG VO, you can indicate this by using local as the VO.
site_policy Url This should be a url pointing to the resource's usage policy
contact String This should be the name of the resource's admin contact
email Email address This should be the email address of the admin contact for the resource
city String This should be the city that the resource is located in
country String This should be two letter country code for the country that the resource is located in.
longitude Number This should be the longitude of the resource. It should be a number between -180 and 180.
latitude Number This should be the latitude of the resource. It should be a number between -90 and 90.

Replace at least the parts in red as appropriate:
;===================================================================
;                       Site Information
;===================================================================

[Site Information]
; The group option indicates the group that the OSG site should be listed in,
; for production sites this should be OSG, for vtb or itb testing it should be
; OSG-ITB
;
; YOU WILL NEED TO CHANGE THIS
group = OSG-ITB

; The host_name setting should give the host name of the CE  that is being
; configured, this setting must be a valid dns name that resolves
;
; YOU WILL NEED TO CHANGE THIS
host_name = ce.opensciencegrid.org

; The resource setting should be set to the same value as used in the OIM
; registration at the goc
;
; YOU WILL NEED TO CHANGE THIS
resource = GC3CE


; The resource_group setting should be set to the same value as used in the OIM
; registration at the goc
;
; YOU WILL NEED TO CHANGE THIS
resource_group = GC3

; The sponsor setting should list the sponsors for your cluster, if your cluster
; has multiple sponsors, you can separate them using commas or specify the
; percentage using the following format 'osg, atlas, cms' or
; 'osg:10, atlas:45, cms:45'
;
; YOU WILL NEED TO CHANGE THIS
sponsor = OSG

; The site_policy setting should give an url that lists your site's usage
; policy
site_policy = UNAVAILABLE

; The contact setting should give the name of the admin/technical contact
; for the cluster
;
; YOU WILL NEED TO CHANGE THIS
contact = user@opensciencegrid.org

; The email setting should give the email address for the technical contact
; for the cluster
;
; YOU WILL NEED TO CHANGE THIS
email = user@opensciencegrid.org

; The city setting should give the city that the cluster is located in
;
; YOU WILL NEED TO CHANGE THIS
city = MyTown

; The country setting should give the country that the cluster is located in
;
; YOU WILL NEED TO CHANGE THIS
country = USA

; The longitude setting should give the longitude for the cluster's location
; if you are in the US, this should be negative
; accepted values are between -180 and 180
;
; YOU WILL NEED TO CHANGE THIS
longitude = 50

; The latitude setting should give the latitude for the cluster's location
; accepted values are between -90 and 90
;
; YOU WILL NEED TO CHANGE THIS
latitude = 50

PBS

This section describes the parameters for a pbs jobmanager if it's being used in the current CE installation. If PBS is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-pbs.ini which is provided by the osg-configure-pbs RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the PBS jobmanager is being used or not.
pbs_location String This should be set to be directory where pbs is installed. osg-configure will try to loocation for the pbs binaries in pbs_location/bin.
job_contact String This should be the contact string for the pbs jobmanager (e.g. my.host.com/jobmanager-pbs)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-pbs-seg. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the PBS SEG module is enabled for the PBS job manager. Although using the SEG requires access to your PBS log files on your CE, it also substantially reduces the load on your CE and PBS scheduler since Globus will be able to monitor job status by examining the PBS logs instead of querying the PBS scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your PBS installation so that the SEG module can monitor jobs running on your cluster.
accounting_log_directory String This setting is used to tell Gratia where to find your accounting log files, and it is required for proper accounting.
pbs_server String This setting is optional and should point to your PBS server node if it is different from your OSG CE

Replace at least the parts in red as appropriate:
;===================================================================
;                              PBS
;===================================================================


[PBS]
; This section has settings for configuring your CE for a PBS job manager

; The enabled setting indicates whether you want your CE to use a PBS job 
; manager
; valid answers are True or False
enabled = FALSE

; The pbs_location setting should give the location of pbs install directory
; On rpm installations, this should be /usr if the pbs binaries (qstat, qsub, etc.) 
; are in /usr/bin
pbs_location = /usr

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-pbs) 
job_contact = host.name/jobmanager-pbs

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and determines whether the PBS SEG module is enabled
; for the PBS job manager. Although using the SEG requires access to your PBS
; log files on your CE, it also substantially reduces the load on your CE and
; PBS scheduler since Globus will be able to monitor job status by examining
; the PBS logs instead of querying the PBS scheduler.
;
; By default, this is disabled
seg_enabled = DEFAULT

; This setting is required if the SEG is enabled. This setting should point
; to the directory with your server log files for your PBS installation so
; that the SEG module can monitor jobs running on your cluster. 
;
; If you enable the SEG, you will need to change this and point it at a valid 
; directory
log_directory = UNAVAILABLE

; This setting is required for Gratia to work. It needs to read the PBS
; accounting log files in order to report accounting data. Common locations are:
; /var/torque/server_priv/accounting
; /var/lib/torque/server_priv/accounting
accounting_log_directory = UNAVAILABLE

; This setting is optional and should point to your PBS server node if it is
; different from your OSG CE 
pbs_server = UNAVAILABLE

; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

LSF

This section describes the parameters for a LSF jobmanager if it's being used in the current CE installation. If LSF is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-lsf.ini which is provided by the osg-configure-lsf RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the LSF jobmanager is being used or not.
lsf_location String This should be set to be directory where lsf is installed
job_contact String This should be the contact string for the lsf jobmanager (e.g. my.host.com/jobmanager-lsf)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-lsf. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the LSF SEG module is enabled for the PBS job manager. Although using the SEG requires access to your LSF log files on your CE, it also substantially reduces the load on your CE and LSF scheduler since Globus will be able to monitor job status by examining the LSF logs instead of querying the scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your LSF installation so that the SEG module can monitor jobs running on your cluster.

Replace at least the parts in red as appropriate:
;===================================================================
;                              LSF
;===================================================================


[LSF]
; This section has settings for configuring your CE for a LSF job manager

; The enabled setting indicates whether you want your CE to use a LSF job 
; manager
; valid answers are True or False
enabled = FALSE

; The lsf setting should give the location of the lsf install directory
lsf_location = DEFAULT

; The lsf_profile setting should give the location of the lsf_profile file
lsf_profile = DEFAULT

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-lsf) 
job_contact = host.name/jobmanager-lsf

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and determines whether the LSF SEG module is enabled
; for the LSF job manager. Although using the SEG requires access to your LSF
; log files on your CE, it also substantially reduces the load on your CE and
; LSF scheduler since Globus will be able to monitor job status by examining
; the LSF logs instead of querying the LSF scheduler.
;
; By default, this is disabled
seg_enabled = DEFAULT

; This setting is required if the SEG is enabled and for gratia accounting. This 
; setting should point to the directory with your server log files for your LSF 
; installation so that the SEG module can monitor jobs running on your cluster 
; and for the gratia probes to get your accounting information. 
;
; If you enable the SEG or gratia accounting, you will need to change this and 
; point it at a valid directory
;
log_directory = UNAVAILABLE


; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

SGE

This section describes the parameters for a SGE jobmanager if it's being used in the current CE installation. If SGE is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-sge.ini which is provided by the osg-configure-sge RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the SGE jobmanager is being used or not.
sge_root String This should be set to be directory where sge is installed (e.g. same as $SGE_ROOT variable).
sge_cell String The sge_cell setting should be set to the value of $SGE_CELL for your SGE install.
job_contact String This should be the contact string for the sge jobmanager (e.g. my.host.com/jobmanager-sge)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-sge. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the SGE SEG module is enabled for the SGE job manager. Although using the SEG requires access to your SGE log files on your CE, it also substantially reduces the load on your CE and SGE scheduler since Globus will be able to monitor job status by examining the SGE logs instead of querying the SGE scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your SGE installation so that the SEG module can monitor jobs running on your cluster.
default_queue String This setting determines queue that jobs should be placed in if the job description does not specify a queue.
available_queues String This setting indicates which queues are available on the cluster and should be used for validation when validate_queues is set.
validate_queues String This setting determines whether the globus jobmanager should check the job RSL and verify that any queue specified matches a queue available on the cluster. If available_queues is set, that list of queues will be used for validation, otherwise SGE will be queried for available queues

Replace at least the parts in red as appropriate:
;===================================================================
;                              SGE
;===================================================================


[SGE]
; This section has settings for configuring your CE for a SGE job manager

; The enabled setting indicates whether you want your CE to use a SGE job 
; manager
; valid answers are True or False
enabled = FALSE

; The sge_root setting should give the location of sge install directory
;
sge_root = UNAVAILABLE

; The sge_cell setting should be set to the value of $SGE_CELL for your SGE
; install.
sge_cell = UNAVAILABLE


; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-sge) 
job_contact = host.name/jobmanager-sge

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

Condor

This section describes the parameters for a Condor jobmanager if it's being used in the current CE installation. If Condor is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-condor.ini which is provided by the osg-configure-condor RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Condor jobmanager is being used or not.
condor_location String This should be set to be directory where condor is installed. If this is set to a blank variable, DEFAULT or UNAVAILABLE, the osg-configure script will try to get this from the CONDOR_LOCATION environment variable if available otherwise it will use /usr which works for the RPM installation.
condor_config String This should be set to be path where the condor_config file is located. If this is set to a blank variable, DEFAULT or UNAVAILABLE, the osg-configure script will try to get this from the CONDOR_CONFIG environment variable if available otherwise it will use /etc/condor/condor_config, the default for the RPM installation.
job_contact String This should be the contact string for the condor jobmanager (e.g. my.host.com/jobmanager-condor)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-condor. If it is set to False or not set then that prefix will not be added.

Replace at least the parts in red as appropriate:
;===================================================================
;                             Condor
;===================================================================


[Condor]
; This section has settings for configuring your CE for an HTCondor job manager

; The enabled setting indicates whether you want your CE to use an HTCondor job
; manager.
; valid answers are True or False
enabled = FALSE

; The condor_location setting should give the location of condor install directory.
; If you are using an RPM installation of HTCondor, set this to DEFAULT.
; Otherwise, set this to the root directory of the HTCondor install,
; such that $CONDOR_LOCATION/bin contains the condor binaries.
condor_location = DEFAULT

; The condor_config setting should give the location of condor config file.
; If you are using an RPM installation of HTCondor, or wish osg-configure to
; detect the correct value, set this to DEFAULT.
; On an RPM install of HTCondor, this file is located at /etc/condor/condor_config.
; On a tarball install of HTCondor, this file is typically located at
; $CONDOR_LOCATION/etc/condor_config.
condor_config = DEFAULT

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-condor) 
job_contact = host.name/jobmanager-condor

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

Slurm

This section describes the parameters for a Slurm jobmanager if it's being used in the current CE installation. If Slurm is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-slurm.ini which is provided by the osg-configure-slurm RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Slurm jobmanager is being used or not.
slurm_location String This should be set to be directory where slurm is installed. osg-configure will try to location for the slurm binaries in slurm_location/bin.
job_contact String This should be the contact string for the pbs jobmanager (e.g. my.host.com/jobmanager-pbs)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-pbs-seg. If it is set to False or not set then that prefix will not be added.
db_host String Hostname of the machine hosting the SLURM database. This information is needed to configure the SLURM gratia probe.
db_port String Port of where the SLURM database is listening. This information is needed to configure the SLURM gratia probe.
db_user String Username used to access the SLURM database. This information is needed to configure the SLURM gratia probe.
db_pass String The location of a file containing the password used to access the SLURM database. This information is needed to configure the SLURM gratia probe.
db_name String Name of the SLURM database. This information is needed to configure the SLURM gratia probe.
slurm_cluster String The name of the Slurm cluster

Replace at least the parts in red as appropriate:
;===================================================================
;                              SLURM
;===================================================================


[SLURM]
; This section has settings for configuring your CE for a Slurm job manager

; The enabled setting indicates whether you want your CE to use a PBS job
; manager
; valid answers are True or False
enabled = FALSE


; The slurm_location setting should give the location of slurm install directory
; On rpm installations, this should be /usr if the slurm binaries (qstat, qsub, etc.)
; are in /usr/bin
slurm_location = /usr

; The job_contact setting should give the contact string for the jobmanager
; on this CE (e.g. host.name/jobmanager-pbs)
;
; Since we're using the PBS emulation in Slurm with the globus jobamanger,
; the jobmanager should be set to pbs
job_contact = host.name/jobmanager-pbs

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and will allow globus to accept limited proxies if set.
; Changing this is not needed for almost all sites. If set to True it will add
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork.
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this
; accept_limited = False


; Hostname of the machine hosting the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_host = host.name

; Port of where the SLURM database is listening.T his information is needed
; to configure the SLURM gratia probe.
db_port = port

; Username used to access the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_user = user

; Location of a file containing the password used to access the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_pass = file/location

; Name of the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_name = database_name

slurm_cluster = cluster_name


Bosco

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Bosco jobmanager is being used or not.
users String A comma separated string. The existing usernames on the CE for which to install Bosco and allow submissions. In order to have separate usernames per VO, for example the CMS VO to have the cms username, each user must have Bosco installed. The osg-configure service will install Bosco on each of the users listed here.
endpoint String The remote cluster submission host for which Bosco will submit jobs to the scheduler. This is in the form of user@example.com, exactly as you would use to ssh into the remote cluster.
batch String The type of scheduler installed on the remote cluster.
ssh_key String The location of the ssh key, as created above.

Replace at least the parts in red as appropriate:
;===================================================================
;                              Bosco
;===================================================================

[BOSCO]
; This section has settings for configuring your CE for a BOSCO job manager

; (Required) Determines whether you want a bosco route for the compute element
enabled = TRUE

; (Required) A comma separated list of users for which to enable bosco submission
users = bosco, cms

; (Required) The endpoint should be the hostname of the remote login node, in the form:
; @
endpoint = bosco@example.com

; (Required) The type of batch system used on the remote cluster.  Possible values are:
; pbs, lsf, sge, condor, slurm
batch = slurm

; (Required) The location of the SSH key that will allow passwordless login to the
; remote cluster's login node.
ssh_key = /etc/osg/bosco.key

; (Optional) The maximum number of jobs to submit to the remote cluster, idle + running.
max_jobs = 1000

Managed Fork

This section determines whether the gatekeeper on your CE uses Trash.ReleaseDocumentationManagedFork to handle local jobs or not. This section is contained in /etc/osg/config.d/15-managedfork.ini which is provided by the osg-configure-managedfork RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the managed fork service is being used or not.
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. If it is set to False or not set then that prefix will not be added.

Replace at least the parts in red as appropriate:
;===================================================================
;                              Managed Fork
;===================================================================

[Managed Fork]
; The enabled setting indicates whether managed fork is in use on the system
; or not. You should set this to True or False
enabled = True

Misc Services

This section handles the configuration of services that do not have a dedicated section for their configuration. This section is contained in /etc/osg/config.d/10-misc.ini which is provided by the osg-configure-misc RPM.

Option Values Accepted Explanation
glexec_location String This gives the location of the glexec installation on the worker nodes, if it is present. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If it is not installed, set this to UNAVAILABLE
gums_host String This setting is used to indicate the hostname of the GUMS host that should be used for authentication. If GUMS is not used, this should be set to UNAVAILABLE
authorization_method gridmap, xacml, local-gridmap This indicates which authentication method your site uses.
edit_lcmaps_db True, False (Optional, default True) If true, osg-configure will make changes to /etc/lcmaps.db to set your authorization method
Replace at least the parts in red as appropriate:
;===================================================================
;                              Misc Services
;===================================================================


[Misc Services]
; If you have glexec installed on your worker nodes, enter the location
; of the glexec binary in this setting
glexec_location = UNAVAILABLE


; This setting should be set to the host used for gums host.  
; If your site is not using a gums host, you can set this to DEFAULT
gums_host = DEFAULT

; This setting should be set to one of the following: gridmap, xacml
; to indicate whether gridmap files or  gums with xacml should be used
authorization_method = gridmap

; As of OSG 1.2.30 cleanup scripts are included to help delete old files
; For more information see the Twiki:
;   https://twiki.grid.iu.edu/bin/view/Trash/ReleaseDocumentationVdtCleanupScripts
; Defaults to disabled
enable_cleanup = FALSE

; The age in days after which files that have not been accessed should be 
; deleted. Jobs that run longer than this number of days may be killed because
; necessary files in VDT_LOCATION/globus/tmp may be removed.  If this is a 
; problem for you do not run the vdt-cleanup script and email 
; vdt-support@opensciencegrid.org
; Default = 14
cleanup_age_in_days = 14

; The user's home directories that should be cleaned up.  The default value
; is @vo-file.  This will open osg-user-vo-map.txt and try to get the list
; of users from this file.  Alternatively you can specify a whitespace 
; separated list of users.
cleanup_users_list = @vo-file

; The cron time at which the cleanup script will run.  By default it will run
; once a day during the night.
cleanup_cron_time = 15 1 * * *

RSV

This section handles the configuration and setup of the RSV services This section is contained in /etc/osg/config.d/30-rsv.ini which is provided by the osg-configure-rsv RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the rsv service is being used or not.
rsv_user String This gives username that rsv will run under. If this is blank or set to UNAVAILABLE, it will default to rsv.
gratia_probes String This settings indicates which rsv gratia probes should be used. It is a list of probes separated by a comma. Valid probes are metric, condor, pbs, lsf, sge, managedfork, hadoop-transfer, and gridftp-transfer
ce_hosts String This option lists the serviceURI of the CEs that generic RSV CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This must be set if enable_ce_probes is set. If this is left blank or set to UNAVAILABLE, the CE's hostname is used by default.
gram_ce_hosts String This option lists the serviceURI of the Globus GRAM-based CEs that the RSV GRAM CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This should be set if gram_gateway_enabled in the Gateway configuration section is True.
htcondor_ce_hosts String This option lists the serviceURI of the HTCondor-CE-based CEs that the RSV HTCondor-CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This should be set if htcondor_gateway_enabled in the Gateway configuration section is True.
gums_hosts String This option lists the serviceURI or FQDN of the CEs or SEs, using GUMS for authentication, that the RSV GUMS probes should check. This should be a list of CE or SE FQDNs (and not a GUMS server FQDN) separated by a comma (e.g. my.host,my.host2,my.host3). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
gridftp_hosts String This option lists the serviceURI of the gridftp servers that the RSV gridftp probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host.iu.edu:2812,my.host2,my.host3). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
gridftp_dir String This should be the directory that the gridftp probes should use during testing. It must be given if enable_gridftp_probes is enabled. This defaults to /tmp if left blank or set to UNAVAILABLE.
srm_hosts String This option lists the serviceURI of the srm servers that the RSV srm probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:8444). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
srm_dir String This should be the directory that the srm probes should use during testing. It must be given if enable_srm_probes is enabled.
srm_webservice_path String This option gives the webservice path that SRM probes need to use along with the host: port. For dcache installations, this should work if left blank. However Bestman-xrootd SEs normally use srm/v2/server as web service path, and so Bestman-xrootd admins will have to pass this option with the appropriate value (for example: "srm/v2/server") for the SRM probes to pass on their SE.
service_cert String This option should point to the public key file (pem) for your service certificate. If this is left blank or set to UNAVAILABLE and the user_proxy setting is set , it will default to /etc/grid-security/rsvcert.pem
service_key String This option should point to the private key file (pem) for your service certificate. If this is left blank or set to UNAVAILABLE and the service_cert setting is enabled, it will default to /etc/grid-security/rsvkey.pem .
service_proxy String This should point to the location of the rsv proxy file. If this is left blank or set to UNAVAILABLE and the use_service_cert setting is enabled, it will default to /tmp/rsvproxy.
legacy_proxy True, False Default=False. If set to true, generate a legacy Globus proxy from the service certificate using grid-proxy-init -old. This has no effect for user proxies since those are not automatically generated by RSV. New in osg-configure-1.0.8
user_proxy String If you don't use a service certificate for rsv, you will need to specify a proxy file that RSV should use in the proxy_file setting. If this is set, then service_cert, service_key, and service_proxy should be left blank, or set to UNAVAILABE or DEFAULT.
enable_gratia True, False This option will enable RSV record uploading to central RSV collector at the GOC. This should be set to True on all OSG resources (and to False on non-OSG resources).
setup_rsv_nagios True, False This option indicates whether rsv should upload results to a local nagios server instance. This should be set to True or False.
This plugin is provided as an experimental component, and admins are recommend not to enable it on production resources.
rsv_nagios_conf_file String The rsv_nagios_conf_file option indicates the location of the rsv nagios file to use for configuration details. This file needs to be configured locally for RSV-Nagios forwarding to work -- see inline comments in file for more information.
condor_location String If you installed Condor in a non-standard location (somewhere other than /usr, which is where the RPM puts it) you must specify the path to the install dir here.

Replace at least the parts in red as appropriate:

Gateway

This section gives information about the options in the Gateway section of the configuration files. These options control the behavior of job gateways on the CE. A CE may be based on Globus GRAM, in which case it would have globus-gatekeeper as the gateway. Or, a CE may be based on HTCondor-CE, in which case it would have condor-ce as the gateway.

Option Values Accepted Explanation
gram_gateway_enabled True, False True if the CE is using Globus GRAM, False otherwise. GRAM Job managers will be configured for enabled batch systems. RSV will use GRAM to launch remote probes.
htcondor_gateway_enabled True, False True if the CE is using HTCondor-CE, False otherwise. HTCondor-CE will be configured to support enabled batch systems. RSV will use HTCondor-CE to launch remote probes.
job_envvar_path String The value of the PATH environment variable to put into HTCondor jobs running with HTCondor-CE. This value is ignored if not using that batch system/gateway combination.

Storage

This section gives information about the options in the Storage section of the configuration file. Several of these values are constrained and need to be set in a way that is consistent with one of the OSG storage models. Please review the Storage Related Parameters section of the Environment Variables description as well as the Overview of Services? and Site Planning discussions for explanations of the various storage models and the requirements for them. This section is contained in /etc/osg/config.d/10-storage.ini which is provided by the osg-configure-ce RPM.

Option Values Accepted Explanation
se_available True, False This indicates whether there is an associated SE available. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node.
default_se String If an SE is available at your cluster, set default_se to the hostname of this SE, otherwise set default_se to UNAVAILABLE. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node.
grid_dir String The grid_dir setting should point to the directory which holds the files from the OSG worker node package. If you have installed the worker node client via RPM (the normal case) it should be /etc/osg/wn-client. If you have somehow installed the worker node in a special location (perhaps via the old Pacman-based VDT in a filesystem shared on all worker nodes), it should be the location of that directory. This directory should be visible on all of the computer nodes and will be accessed via the $OSG_GRID environment variable. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. Read access is required, though worker nodes don't need write access.
app_dir String The app_dir setting should point to the directory which contains the VO specific applications, this should be visible on both the CE and worker nodes and will be accessed via $OSG_APP. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. Only the CE needs to have write access to this directory. The $OSG_APP directory must also contain a sub-directory (etc) with 1777 permissions.
data_dir String The data_dir setting should point to a directory that can be used to store and stage data in and out of the cluster and accessed via the $OSG_DATA variable. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. This directory should be readable and writable on both the CE and worker nodes.
worker_node_temp String The worker_node_temp directory (accessed via $OSG_WN_TMP) should point to a directory that can be used as scratch space on compute nodes. It should allow read and write access on a worker node and can be visible to just that worker node. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE
site_read String The site_read setting should be the location or url to a directory that can be read to stage in data via the variable $OSG_SITE_READ. This is an url if you are using a SE. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE
site_write String The site_write setting should be the location or url to a directory that can be write to stage out data via the variable $OSG_SITE_WRITE. This is an url if you are using a SE . Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE

Replace at least the parts in red as appropriate:
;===================================================================
;                            Storage 
;===================================================================

[Storage]
;
; Several of these values are constrained and need to be set in a way
; that is consistent with one of the OSG storage models
;
; Please refer to the OSG release documentation for an indepth explanation 
; of the various storage models and the requirements for them

; If you have a SE available for your cluster and wish to make it available 
; to incoming jobs, set se_available to True, otherwise set it to False
se_available = FALSE

; If you indicated that you have an se available at your cluster, set default_se to
; the hostname of this SE, otherwise set default_se to UNAVAILABLE
default_se = UNAVAILABLE

; The grid_dir setting should point to the directory which holds the files 
; from the OSG worker node package, it should be visible on all of the computer
; nodes (read access is required, worker nodes don't need to be able to write) 
; 
; For RPM installations, /etc/osg/wn-client is the correct location
grid_dir = /etc/osg/wn-client/

; The app_dir setting should point to the directory which contains the VO 
; specific applications, this should be visible on both the CE and worker nodes
; but only the CE needs to have write access to this directory
; 
; YOU WILL NEED TO CHANGE THIS
app_dir = /osg/app

; The data_dir setting should point to a directory that can be used to store 
; and stage data in and out of the cluster.  This directory should be readable
; and writable on both the CE and worker nodes
; 
; YOU WILL NEED TO CHANGE THIS
data_dir = UNAVAILABLE

; The worker_node_temp directory should point to a directory that can be used 
; as scratch space on compute nodes, it should allow read and write access on the 
; worker nodes but can be local to each worker node
; 
; YOU WILL NEED TO CHANGE THIS
worker_node_temp = /scratch

; The site_read setting should be the location or url to a directory that can 
; be read to stage in data, this is an url if you are using a SE 
; 
; YOU WILL NEED TO CHANGE THIS
site_read = UNAVAILABLE

; The site_write setting should be the location or url to a directory that can 
; be write to stage out data, this is an url if you are using a SE 
; 
; YOU WILL NEED TO CHANGE THIS
site_write = UNAVAILABLE

Squid

This section handles the configuration and setup of the squid web caching and proxy service. This section is contained in /etc/osg/config.d/01-squid.ini which is provided by the osg-configure-squid RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the squid service is being used or not.
location String This should be set to the hostname:port of the squid server.

Replace at least the parts in red as appropriate:
;===================================================================
;                             Squid
;===================================================================

[Squid]
; Set the enabled setting to True if you have squid installed and wish to 
; use it, otherwise set it to False 
enabled = FALSE

; If you are using squid, specify the location of the squid server in the 
; location setting, this should be an url
location = DEFAULT

; If you are using squid, use the policy setting to indicate which cache
; replacement policy squid is using
policy = DEFAULT

; If you are using squid, use the cache_size setting to indicate which the 
; size of the disk cache that squid is using
cache_size = DEFAULT

; If you are using squid, use the memory_size setting to indicate which the 
; size of the memory cache that squid is using
memory_size = DEFAULT

GIP

The Generic Information Providers (GIP) use multiple section of the OSG configuration files:

  • GIP: generic options
  • Subcluster*: options about homogeneous subclusters
  • Resource Entry*: options for specifying ATLAS queues for AGIS
  • SE*: options about storage elements

Notes for multi-CE sites.

If you would like to properly advertise multiple CEs per cluster, make sure that you:
  • Set the value of cluster_name in the [GIP] section to be the same for each CE.
  • Set the value of other_ces in the [GIP] section to be the hostname of the other CEs at your site; this should be comma separated. So, if you have two CEs, ce1.example.com and ce2.example.com, the value of other_ces on ce1.example.com should be "ce2.example.com". This assumes that the same queues are visible on each CE.
  • Set the value of site_name in the "Site Information" section to be the same for each CE.
  • Have the exact same configuration values for the GIP, SE*, and Subcluster* sections in each CE.

It is good practice to run "diff" between the config.ini of the different CEs. The only changes should be the value of localhost in the [DEFAULT] section and the value of other_ces in the [GIP] section.

GIP Section

The options given in bold type are mandatory.

Option Values Accepted Explanation
advertise_gums True, False Defaults to False. If you want GIP to query and advertise your gums server set this to True.
advertise_gsiftp True, False Defaults to True. If you don't want GIP to advertise your gridftp server set this to False.
gsiftp_host String This should be set to the name of the gridftp server GIP will advertise if the advertise_gridftp setting is set to True.
cluster_name String This should only be set if you run multiple gatekeepers for the same cluster; if you do, set this value to the FQDN of the head node of the cluster.
other_ces String This should only be set if you run multiple gatekeepers for the same cluster; if you do, set this value to the comma-separate list of FQDNs for the other CEs at this site.

GIP Subcluster Configuration

Each homogeneous set of worker node hardware is called a subcluster. For each subcluster in your cluster, fill in the information about the worker node hardware by creating a new Subcluster section with a unique name in the following format: [Subcluster CHANGEME] where CHANGEME is the globally unique subcluster name (yes, it must be a globally unique name for the whole grid, not just unique to your site. Get creative).

At least one subcluster section is required; please populate the information for all your subclusters. For WLCG sites, information filled in here will be advertised as part of your MoU commitment, so please strive to make sure it is correct. The information needed is:

Option Values Accepted Explanation
name String The same name that is in the Section label; it should be globally unique
node_count Positive Integer Number of worker nodes in the subcluster
ram_mb Positive Integer Megabytes of RAM per node
cpu_model String CPU model, as taken from /proc/cpuinfo; please, no abbreviations!
cpu_vendor String Vendor's name: AMD, Intel, or ??
cpu_speed_mhz Positive Integer Approximate speed, in MHZ, of the chips
cpus_per_node Positive Integer Number of CPUs (physical chips) per node
cores_per_node Positive Integer Number of cores per node
inbound_network True, False True if external hosts can reach the worker nodes via their hostnames
outbound_network True, False True if worker nodes in this subcluster can communicate with the external world
cpu_platform x86_64, i686 CPU architecture

In addition, there are several attributes that are only used on HTCondor-CE based CEs:

allowed_vos String List Comma-separated list of the VOs allowed to run jobs on this subcluster
max_wall_time Positive Integer Maximum wall-clock time, in minutes, a job is allowed to run on this subcluster
queue String The queue to which jobs should be submitted in order to run on this subcluster
extra_transforms Classad Transformation attributes which the HTCondor Job Router should apply to incoming jobs so they can run on this subcluster

This page shows the mapping from these attribute names to GLUE attribute names

GIP Resource Entry Configuration (ATLAS only)

If you are configuring a CE for the ATLAS VO, you must provide hardware information to advertise the queues that are available to AGIS. For each queue, create a new Resource Entry section with a unique name in the following format: [Resource Entry RESOURCE] where RESOURCE is a globally unique resource name (it must be a globally unique name for the whole grid, not just unique to your site). The following options are required for the Resource Entry section and are used to generate the data required by AGIS:

Option Values Accepted Explanation
name String The same name that is in the Resource Entry label; it must be globally unique
max_wall_time Positive Integer Maximum wall-clock time (in minutes) that a job is allowed to run on this resource
queue String The queue to which jobs should be submitted to run on this resource
cpucount (alias cores_per_node) Positive Integer Number of cores that a job using this resource can get
maxmemory (alias ram_mb) Positive Integer Maximum amount of memory (in MB) that a job using this resource can get

The following attributes are optional:

Option Values Accepted Explanation
allowed_vos Comma-separated List The VOs that are allowed to run jobs on this resource
subclusters Comma-separated List The physical subclusters the resource entry refers to; must be defined elsewhere in the file
vo_tag String An arbitrary label that is added to jobs routed through this resource

GIP SE Configuration

For each storage element, add a new section called [SE_CHANGEME] where CHANGEME is the name of your SE. Each SE name must be unique for the entire grid, so make sure to not pick anything generic like "MAIN". Each SE section must start with the words "SE", and cannot actually be named "CHANGEME".

SE configuration can be a bit tricky. We provide a few examples here.?

Generic SE

We first outline the configuration for a generic SE, then have additional comments for BestMan and dCache SEs. For WLCG sites to be advertised correctly, they must use the bestman or dcache19 provider.:

Option Values Accepted Explanation
enabled Boolean True/False Indicates whether or not this SE section is enabled for the GIP.
name String Name of the SE as registered in OIM.
srm_endpoint String The endpoint of the SE. It MUST have the hostname, port, and the server location (such as /srm/v2/server). It MUST NOT have the ?SFN= string. Example: srm://srm.example.com:8443/srm/v2/server
provider_implementation String Set to static for a generic SE; we recommend BestMan sites use the bestman implementation and dCache sites use dcache or dcache19. See notes below
implementation String Name of the SE implementation.
version String Version number of the SE implementation. Some provider implementations will attempt to auto-detect this.
default_path String Default storage paths for the VOs; VONAME is replaced with the VO's name.
vo_dirs Comma-separated string A comma-separated list of VONAME:PATH pairs; the PATH will override the default_path attribute for VONAME.
allowed_vos Comma-separated list of VOs By default, all VOs are advertised as having access to your SE. If you want only a subset of VOs to be advertised, list them here

BestMan and dCache notes

For BestMan-based SEs, there are a few additional comments to make and extra options:

Option Values Accepted Explanation
srm_endpoint String The endpoint of the SE. It is most likely of the form srm://srm.example.com:8443/srm/v2/server.
provider_implementation String Set to 'bestman'
implementation String Set to 'bestman'
version String Version number; the "bestman" provider implementation will attempt to auto-detect this also.
use_df True, False Set to True if the bestman provide can use 'df' on the directory referenced above to get the freespace information. If set to false, it probably won't detect the correct info. Use this if BestMan writes into a file system which is mounted on your head node (such as xrootdfs, IBRIX, or GPFS).
The BestMan provider will use srm-ping to query your BestMan instance for information. This means that the user tomcat will need to read a valid hostcert in /etc/grid-security/http/httpcert.pem and hostkey in /etc/grid-security/http/httpkey.pem. BestMan server does not require any authorization to perform srmPing, so you do not need to change anything on your SRM server.

For dCache-based SEs:

Option Values Accepted Explanation
srm_endpoint String The endpoint of the SE. It is most likely of the form srm://srm.example.com:8443/srm/managerv2.
provider_implementation String Set to 'dcache' for dcache 1.8 (additional config is required), 'dcache19' for dcache 1.9, or 'static' for default values. If you use the dcache provider with dCache 1.8, see this page to complete installation. If you use the dcache19 provider, you must fill in the location of your dCache's information provider
infoprovider_endpoint String Url to the dcache information provider. Only necessary for the dcache19 provider.
implementation String Set to 'dcache'
version String dCache version; non-static providers will also attempt to auto-detect this. This should be the version from the output of "rpm -q dcache-server".
srm_host String This most likely does not need editing. If your SRM endpoint has multiple aliases, the GIP might get confused when resolving the IP addresses given by dCache. This setting will force the selection of a specific hostname.
dCache version 1.9 added a new read-only XML information service which feeds the dcache19 provider. The location of the info provider is most likely:
http://dcache_head_node.example.com:2288/info
It runs on the same node that the dcache web interface does. Check your dCache documentation on how to enable the info service if that link does not work.

Since dCache 1.8 does not have a machine-readable information service, the "dcache" provider must access it through the admin interface; this process is documented here. Some site admins consider this a security concern.

SE Space configuration.

Both the dCache and BestMan providers have a concept of a logical storage grouping, or space. You can control how the space access is advertise through adding options to the SE's section. For dCache, a space may be one of several dCache concepts:
  • A link group is a space, if link groups are configured.
  • A pool group is a space.
If a pool group is inside a link group, only the link group will be advertised. The space name is the name of the link group or pool group.

For BestMan, a space is a SRM space token. The name of the space is the space token description.

The following space-related options are valid; replace NAME with the space's name:

Option Values Accepted Explanation
spaces Comma-delimited list Comma-delimited list of all the spaces that should be restricted by VO
space_NAME_vos Comma-delimited list of VOs By default, all VOs are allowed to access all spaces. If you would like to change this for a specific space, "NAME", this should be a comma-delimited list of VOs allowed to access NAME.
space_NAME_default_path String The default storage path for VOs for the space NAME. As in the "default_path" option, the word VONAME is evaluated to be the VO's name.
space_NAME_path Comma-delimited list A list of VONAME:PATH pairs for this space; works like the SE's vo_dirs variable.

These options are not required for BestMan or dCache. Both providers will attempt to guess the correct values for spaces, but are not always perfect.

No SE - Publish CE to SE Bindings for external SE

In very rare cases, a site may not host an SE, but have an agreement with another site to share or otherwise use their SE. For this use case, edit /etc/gip/gip.conf (create if not present). Make sure the following lines are present:
[cesebind]
simple=False
se_list=SE1.example.com,SE2.example.com,...
The se_list is a comma delimited list of SE Unique ID's as configured at the external site. Usually, the SE unique ID is set to the full hostname of the SRM endpoint.

Example Configurations for SE's and Subclusters

Examples taken from working installations of the new style Storage Element and Subcluster sections can be found at ConfigurationFileGIPExamples.

Replace at least the parts in red as appropriate:
;===================================================================
;                              GIP
;===================================================================

[GIP]

; ========= These settings must be changed ==============

;; This setting indicates the batch system that GIP should query
;; and advertise
;; This should be the name of the batch system in lowercase
batch = DEFAULT
;; Options include: pbs, lsf, sge, or condor

; ========= These settings can be left as is for the standard install ========

;; This setting indicates whether GIP should advertise a gsiftp server
;; in addition to a srm server, if you don't have a srm server, this should
;; be enabled
;; Valid options are True or False
advertise_gsiftp = TRUE

;; This should be the hostname of the gsiftp server that gip will advertise
gsiftp_host = DEFAULT

;; This setting indicates whether GIP should query the gums server.
;; Valid options are True or False
advertise_gums = FALSE


;===================================================================
;                          Subclusters
;===================================================================

; For each subcluster, add a new subcluster section.
; Each subcluster name must be unique for the entire grid, so make sure to not
; pick anything generic like "MAIN".  Each subcluster section must start with
; the words "Subcluster", and cannot be named "CHANGEME".

; There should be one subcluster section per set of homogeneous nodes in the
; cluster.

; This data is used for our statistics collections in the OSG, so it's important
; to keep it up to date.  This is important for WLCG sites as it will be used
; to determine your progress toward your MoU commitments!

; If you have many similar subclusters, then feel free to collapse them into
; larger, approximately-correct groups.

; See example below:

[Subcluster T1]
; should be the name of the subcluster
name = SUBCLUSTER_T1
; number of homogeneous nodes in the subcluster
node_count = 10
; Megabytes of RAM per node.
ram_mb = 2024
; CPU model, as taken from /proc/cpuinfo.  Please, no abbreviations!
;#GenuineIntel
;#cpu family    : 6
;#model         : 6
;#model name    : QEMU Virtual CPU version 0.9.1
;#stepping      : 3
;#cpu MHz               : 2660.068
cpu_model = QEMU Virtual CPU version 0.9.1
; Should be something like:
; cpu_model = Dual-Core AMD Opteron(tm) Processor 2216
; Vendor's name -- AMD or Intel?
cpu_vendor = Intel
; Approximate speed, in MHZ, of the chips
cpu_speed_mhz = 2660
; Must be an integer.  Example: cpu_speed_mhz = 2400
; Platform; x86_64 or i686
cpu_platform = x86_64
; Number of CPUs (physical chips) per node
cpus_per_node = 1
; Number of cores per node.
cores_per_node = 2
; For a dual-socket quad-core, you would put cpus_per_node=2 and
; cores_per_node=8

; Set to true or false depending on inbound connectivity.  That is, external
; hosts can contact the worker nodes in this subcluster based on their hostname.
inbound_network = FALSE
; Set to true or false depending on outbound connectivity.  Set to true if the
; worker nodes in this subcluster can communicate with the external internet.
outbound_network = TRUE

; Non-mandatory attributes
; The amount of swap per host in MB
;  swap_mb = 4000
; The per-core SpecInt 2000 score.  This is usually computed for you.
;  SI00 = 2000
; The per-core SpecFloat 2000 score.  This is usually computed for you
;  SF00 = 2000

; A list of VOs that are allowed to submit to this subcluster.  Leave blank
; to allow all VOs to submit.
;allowed_vos = vo1, vo2
; The maximum wall-clock time a job is allowed to run on this subcluster,
; in minutes.  Leave blank or set to 0 to indicate no wall time limit on jobs.
;max_wall_time = 1440
; The queue which jobs should be submitted to in order to run on this resource.
; Equivalent to the HTCondor grid universe classad attribute "remote_queue"
;queue = blue
; Transformation attributes which the HTCondor job router should apply to
; incoming jobs so they can run on this resource, as per
; http://research.cs.wisc.edu/htcondor/manual/v8.3/5_4HTCondor_Job.html
; These are in HTCondor classad syntax, and should be used sparingly.
;extra_transforms = [ set_WantRHEL6 = 1; ]

; Mandatory for WLCG reporting
; The per-core HEPSPEC score.  See your VO representative for more information.
;  HEPSPEC = 8
; The conversion factor from HEPSPEC to SI2K is accepted to be 250.



;===================================================================
;                             SE
;===================================================================

; For each storage element, add a new SE section.
; Each SE name must be unique for the entire grid, so make sure to not
; pick anything generic like "MAIN".  Each SE section must start with
; the words "SE", and cannot be named "CHANGEME".

; There are two main configuration types; one for dCache, one for BestMan

; Don't forget to change the section name!  One section per SE at the site.
[SE CHANGEME]

; The first part of this section shows options which are mandatory for all SEs.
; dCache and BestMan-specific portions are shown afterward.

; Set to False to turn off this SE
enabled = False

; Name of the SE; set to be the same as the OIM registered name
name = SE_CHANGEME
; The endpoint of the SE.  It MUST have the hostname, port, and the server
; location (/srm/v2/server in this case).  It MUST NOT have the ?SFN= string.
srm_endpoint = httpg://srm.example.com:8443/srm/v2/server
; dCache endpoint template: httpg://srm.example.com:8443/srm/managerv2

; How to collect data; the most generic implementation is called "static"
provider_implementation = static
; WLCG sites with a SE *must* use bestman, dcache, or dcache19
; Implementation and version of your SRM SE; usually dcache or bestman
implementation = bestman
; Version refers to the SE version, not the SRM version.
version = 2.2.1.foo
; dCache example: version = 1.9.1
; Default paths for all of your VOs; VONAME is replaced with the VO's name.
default_path = /mnt/bestman/home/VONAME
; Set a specific path for VOs which don't use the default path.
; Comma-separated list of VO:PATH pairs.  Not required.
; vo_dirs=cms:/mnt/bestman/cms, dzero:/mnt/bestman2/atlas

; If your SE provides a POSIX-like mount on your worker nodes, uncomment
; the following line:
; mount_point = /,/
; POSIX-like file systems include Lustre, NFS, HDFS, xrootdfs
; dCache site should not uncomment the line
; The value of `mount_point` should be two paths; first, the path where the
; file system is mounted on the worker nodes, followed by the exported directory
; of the file system.  If you mount your file system on the worker nodes with
; the following command:
;    mount -t nfs nfs.example.com:/exported/dir /mnt/nfs
; then mount_point should look like this:
;    mount_point = /mnt/nfs,/exported/dir

; By default, all VOs are advertised as having access to your SE. If you want only a subset of
; VOs to be advertised, list them as a comma separated list in the allowed_vos setting
; allowed_vos =

; For BestMan-based SEs, uncomment and fill in the following.
;  provider_implementation = bestman
;  implementation = bestman

; Set to TRUE if the bestman provide can use 'df' on the directory referenced
; above to get the freespace information.  If set to false, it probably won't
; detect the correct info.
;  use_df = True

; For dCache-based SEs, uncomment and fill in the following
; How to collect data; set to 'dcache' for dcache 1.8 (additional config req'd
; for this case), 'dcache19' for dcache 1.9, or 'static' for default values.
; provider_implementation = dcache19
;  implementation = dcache
;  If you use the dcache provider, see
; http://twiki.grid.iu.edu/bin/view/InformationServices/DcacheGip
; If you use the dcache19 provider, you must fill in the location of your
; dCache's information provider:
;  infoprovider_endpoint = http://dcache.example.com:2288/info
; SE implementation name; leave as 'dcache'

; Here are working configs for BestMan and dCache
; [SE dCache]
; name = T2_Nebraska_Storage
; srm_endpoint = httpg://srm.unl.edu:8443/srm/managerv2
; provider_implementation = static
; implementation = dcache
; version = 1.8.0-15p6
; default_path = /pnfs/unl.edu/data4/VONAME

; [SE Hadoop]
; name = T2_Nebraska_Hadoop
; srm_endpoint = httpg://dcache07.unl.edu:8443/srm/v2/server
; provider_implementation = bestman
; implementation = bestman
; version = 2.2.1.2.e1
; default_path = /user/VONAME

Local Settings

This section differs from other sections in that there are no set options in this section. Rather, the options set in this section will be placed in the osg-local-job-environment.conf verbatim. The options in this section are case sensitive and the case will be preserved when they are converted to environment variables. The osg-local-job-environment.conf file gets sourced by jobs run on your cluster so any variables set in this section will appear in the environment of jobs run on your system.

Adding a line such as My_Setting = my_Value would result in the an environment variable called My_Setting set to my_Value in the job's environment. my_Value can also be defined in terms of an environment variable (i.e My_Setting = $my_Value) that will be evaluated on the worker node. For example, to add a variable MY_PATH set to /usr/local/myapp, you'd have the following:

[Local Settings]

MY_PATH = /usr/local/myapp

This section is contained in /etc/osg/config.d/40-localsettings.ini which is provided by the osg-configure-ce RPM.

Replace at least the parts in red as appropriate:
;
; Put any environment variables that should go into the local
; job environment for grid jobs here.  The format is
;   env = setting
; and case is preserved, the setting does not need to be quoted
; See https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/ConfigurationFileHelp#Local_Settings
; for more details
[Local Settings]

Info Services

OSG-Info-Services is configured in this section, which is found in 30-infoservices.ini, installed by the osg-configure-infoservices RPM. OSG-Info-Services replaces CEMon in OSG 3.2 and later.

Option Values Accepted Explanation
enabled True, False, Ignore True if OSG-Info-Services should be configured and enabled.
ress_servers String Set to the ReSS servers the installation should report to. Leaving this at the default will report to the OSG Production servers. Note that the OSG ITB ReSS servers have been removed, so even ITB sites should report to the production servers.
bdii_servers String Set to the BDII servers the installation should report to. Leaving this at the default will report to the OSG Production servers. Note that the OSG ITB BDII servers have been removed, so even ITB sites should report to the production servers.
ce_collectors String The server(s) HTCondor-CE information should be sent to. Only available on OSG 3.2 and later, when HTCondor-CE is being used. Setting this to PRODUCTION will report to the OSG Production servers; setting this to ITB will report to the OSG ITB servers; otherwise, set to the hostname:port of a host running a condor-ce-collector daemon.

Replace at least the parts in red as appropriate:
;===================================================================
;                         Info Services
;===================================================================
;
; This configuration file is a replacement for 30-cemon.ini, used
; for osg-info-services. You may have previous settings for CEMon
; that you wish to migrate. You will find them in either
; 30-cemon.ini or 30-cemon.ini.rpmsave.

[Info Services]

; Default servers for production and itb OSG ReSS/BDII servers, please don't touch

; The current production OSG servers
; Production ReSS server
osg-ress-servers = https://osg-ress-1.fnal.gov:8443/ig/services/CEInfoCollector[OLD_CLASSAD]
; Production BDII server
osg-bdii-servers = http://is1.grid.iu.edu:14001[RAW], http://is2.grid.iu.edu:14001[RAW]



; The enable option indicates whether the OSG info services should be enabled or
; disabled.  It should be set to True or False
;
; You generally want OSG info services enabled for any CE installation
enabled = TRUE

; This setting indicates which servers ReSS information should
; be sent to.  Most sites should use the %(osg-ress-servers)s
; setting so that the predefined variable giving the default
; OSG production servers will be used
;
; The server list should be formatted as follows:
; server_uri[format],server_uri[format]
; see the variables at the top of this section for examples of this
ress_servers = DEFAULT

; This setting indicates which servers BDII information should
; be sent to.  Most sites should use the %(osg-bdii-servers)s
; setting so that the predefined variable giving the default
; OSG production servers will be used
;
; The formatting for this are the same as the ress_servers setting
bdii_servers = DEFAULT

; This setting indicates which servers HTCondor-CE information
; should be sent to. These servers should be running a
; condor-ce-collector daemon. Set this to a comma-separated
; list of hostname:port combinations. The default collector port
; 9619 will be used if not specified. You can also use one of these
; special values:
; 'PRODUCTION' : OSG production servers
; 'ITB'        : OSG testing servers
; 'DEFAULT'    : OSG testing servers if this is an ITB site,
;                production otherwise
ce_collectors = DEFAULT

CEMon

CEMon is configured in this section. CEMon is deprecated in OSG 3.1 and replaced by OSG-Info-Services in OSG 3.2.

Please note that setting ress_servers and bdii_servers to %(default)s or %(unavailable)s in the default configuration file will result in the correct servers being used for your reporting. If you need to specify additional servers, the %(osg-ress-servers)s, %(osg-bdii-servers)s, %(itb-ress-servers)s=, or %(itb-bdii-servers)s variables are defined in the default configuration files to make it easier to specify that CEMon should also report to the the various OSG collectors as well. This section is contained in /etc/osg/config.d/30-cemon.ini which is provided by the osg-configure-cemon RPM.

Option Values Accepted Explanation
enabled True , False, Ignore This should be set to True if CEMon should be configured and enabled on the installation being configured.
ress_servers String This should be set to the ReSS servers that the installation should report information to. This should be formatted as server_url[FORMAT],server_url2[FORMAT] where FORMAT is the CEMon format to use for reporting. The format for is OLD_CLASSAD for most ReSS servers. E.g. for production resources this might be https://osg-ress-1.fnal.gov:8443/ig/services/CEInfoCollector[OLD_CLASSAD]
bdii_servers String This should be set to the BDII servers that the installation should report information to. This should be formatted as server_url[FORMAT],server_url2[FORMAT] where FORMAT is the CEMon format to use for reporting. The format is RAW for most BDII servers. E.g. for production resources this might be http://is1.grid.iu.edu:14001[RAW], http://is2.grid.iu.edu:14001[RAW]

Replace at least the parts in red as appropriate:
;===================================================================
;                         CEMon
;===================================================================

[Cemon]

; Default servers for production and itb OSG ress/bdii servers, please don't touch

; The current production osg servers
; Production ReSS server
osg-ress-servers = https://osg-ress-1.fnal.gov:8443/ig/services/CEInfoCollector[OLD_CLASSAD]
; Production BDII server
osg-bdii-servers = http://is1.grid.iu.edu:14001[RAW], http://is2.grid.iu.edu:14001[RAW]

; The current itb osg servers
; ITB ReSS server
itb-ress-servers = https://osg-ress-4.fnal.gov:8443/ig/services/CEInfoCollector[OLD_CLASSAD]
; ITB BDII server
itb-bdii-servers = http://is1.grid.iu.edu:14001[RAW], http://is2.grid.iu.edu:14001[RAW]


; The enable option indicates whether cemon should be enabled or
; disabled.  It should be set to True or False
;
; You generally want Cemon enabled for any CE installation
enabled = TRUE

; This setting indicates which servers ress information should
; be sent to.  Most sites should use the %(osg-ress-servers)s
; setting so that the predefined variable giving the default
; osg production servers will be used, ITB admins can use
; %(itb-ress-servers)s for default itb servers
;
; The server list should be formated as follows:
; server_uri[format],server_uri[format]
; see the variables at the top of this section for examples of this
ress_servers = %(itb-ress-servers)s

; This setting indicates which servers bdii information should
; be sent to.  Most sites should use the %(osg-bdii-servers)s
; setting so that the predefined variable giving the default
; osg production servers will be used, ITB admins can use
; %(itb-bdii-servers)s for default itb servers
; 
; The formatting for this are the same as the ress_servers setting
bdii_servers = %(itb-bdii-servers)s

Gratia

This section configures Gratia. Like with the CEMon configuration, if probes is set to UNAVAILABLE, then osg-configure will appropriate default values. If you need to specify custom reporting (e.g. a local gratia collector) in addition to the default probes, %(osg-jobmanager-gratia)s, %(osg-gridftp-gratia)s, %(osg-metric-gratia)s, %(itb-jobmanager-gratia)s, %(itb-gridftp-gratia)s, %(itb-metric-gratia)s are defined in the default configuration files to make it easier to specify the standard osg reporting. This section is contained in /etc/osg/config.d/30-gratia.ini which is provided by the osg-configure-gratia RPM.

Option Values Accepted Explanation
enabled True , False, Ignore This should be set to True if gratia should be configured and enabled on the installation being configured.
resource String This should be set to the resource name as given in the OIM registration
probes String This should be set to the gratia probes that should be enabled. A probe is specified by using as [probe_type]:server:port . Valid probe_types are metric (for rsv), jobmanager (for the appropriate jobmanager probe) and gridftp for the gridftp-transfer-probe.

Replace at least the parts in red as appropriate:
;===================================================================
;                         Gratia
;===================================================================

[Gratia]

; Default gratia servers
;
; Please don't change these unless you have good reason to do so

; Variables for osg itb probes, CE installations should use the
; jobmanager probe
itb-jobmanager-gratia = jobmanager:gratia-osg-itb.opensciencegrid.org:80
itb-gridftp-gratia = gridftp:gratia-osg-itb.opensciencegrid.org:80


; Variables for osg production probes, CE installations should use the
; jobmanager probe
osg-jobmanager-gratia = jobmanager:gratia-osg-prod.opensciencegrid.org:80
osg-gridftp-gratia = gridftp:gratia-osg-transfer.opensciencegrid.org:80


; The enable option indicates whether gratia should be enabled or
; disabled.  It should be set to True or False
;
; You generally want Gratia enabled for any CE installation, in
; addition SE installations may want to enable gratia to use
; the gratia gridftp reporting
enabled = TRUE


; This setting specifies the resource that gratia will use to report
; accounting information, on a CE if you leave this blank, gratia will
; use the resource setting from the Site Information section
;
resource = DEFAULT

; This setting indicates which probes should be enabled for gratia
; The list should be given as probe_name1:host1:port1, probe_name2:host2:port2
; where probe_name is either jobmanager or gridftp
; host is a fully qualified domain name
; port is the port that the server is listening on
; CEs should have entries for jobmanager probe
; SEs should use gridftp if they would like to enable gridftp transfer
; accounting
;
; for convenience admins can use %(osg-jobmanager-gratia)s, and
; %(osg-gridftp-gratia)s for production sites
; and %(itb-jobmanager-gratia)s, and %(itb-gridftp-gratia)s for ITB sites
probes = %(itb-jobmanager-gratia)s,%(itb-gridftp-gratia)s

Network

This section is used to configure the ports that Globus should use in order to work with firewalls that may be present on your site. This section is used to configure shell profile files and other configuration files needed to allow Globus tools to work correctly in the presence of a firewall. This section is contained in /etc/osg/config.d/40-network.ini which is provided by the osg-configure-network RPM.

Option Values Accepted Explanation
source_range String This setting should list the range of ports that inbound connections to Globus may arrive on. It should be formatted as low_port,high_port (e.g. 1024,2048)
source_state_file String This setting should give the file that Globus should use to track ports that it has used for inbound connections
port_range String This setting should list the range of ports that Globus should use for outbound connections. It should be formatted as low_port,high_port (e.g. 1024,2048)
port_state_file String This setting should give the file that Globus should use to track ports that it has used for outbound connections

Replace at least the parts in red as appropriate:
;===================================================================
;                              Network
;===================================================================

[Network]
; For more discussion on what other things may need to be set to work 
; with the globus_source_range and globus_port_range, see
; https://twiki.grid.iu.edu/bin/view/Documentation/Release3/FirewallInformation

; This setting should give the low port and high port for a contiguous range
; of ports that inbound connections to globus may arrive on
; the range should be given as low_port,high_port (e.g. 2048,4096)  
source_range = UNAVAILABLE

; This setting should give the location of the file that globus should
; use to track inbound port usage 
source_state_file = UNAVAILABLE

; This setting should give the low port and high port for a contiguous range
; of ports that outbound connections from globus should use
; the range should be given as low_port,high_port (e.g. 2048,4096)  
port_range = UNAVAILABLE


; This setting should give the location of the file that globus should
; use to track outbound port usage 
port_state_file = UNAVAILABLE

Comments

In "Conventions" shouldn't we say that all mandatory options have a default loaded by the RPM? JamesWeichel 09 Nov 2011 - 15:55
Should the verify command be included for use after the admin edits these files like shown in https://twiki.grid.iu.edu/bin/view/Trash/ReleaseDocumentationConfigureOSGAttributes#Verify_the_Configuration_File JamesWeichel 09 Nov 2011 - 16:01

Topic revision: r42 - 07 Feb 2017 - 19:40:51 - BrianBockelman
Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..