Ini Configuration Options

About this Document

hand This document outlines the settings and options found in the ini files for system administers that are installing and configuring osg software.

This page gives an overview of the options for each of the sections of the configuration files that osg-configure uses.

Conventions

In the tables below:

  • Mandatory options for a section are given in bold type. Sometime the default value may be OK and no edit required, but the variable has to be in the file.
  • Options that are not found in the default ini file are in italics.

Introduction

osg-configure and the INI files in /etc/osg/config.d allow a higl level configuration of OSG services.

Layout

The configuration files used by osg-configure are the one supported by Python's SafeConfigParser, similar in format to the INI configuration file used by MS Windows. Long lines can be split up using continuations as specified in email RFC 822 (each White Space Character can be preceded by a Newline to fold/continue the fileld on a new line). A config file is separated into sections with each section starting with a section name in square brackets (e.g. [Section 1]) . Lines that begin with a ; or a # are treated as comments and ignored. Options can be set using either name : value or name=value pairs. Finally, variables and variable substitutions are supported.

osg-configure reads and uses all of the files in /etc/osg/config.d that have a "ini" suffix. The files in this directory are ordered with a numeric prefix with higher numbers being applied later and thus having higher precedence (e.g. 00-foo.ini has a lower precedence than 99-local-site-settings.ini). Configuration sections and options can be specified multiple times in different files. E.g. a section called [PBS] can be given in 20-pbs.ini as well as 99-local-site-settings.ini.

Each of the files are successively read and merged to create a final configuration that is then used to configure OSG software. Options and settings in files read later override the ones in previous files. This allows admins to create a file with local settings (e.g. 99-local-site-settings.ini) that can be read last and which will be take precedence over the default settings in configuration files installed by various RPMs and which will not be overwritten if RPMs are updated.

Variables and variable substitution

The osg-configure parser allows variables to be defined and used in the configuration file. Any option set in a given section can be used as a variable in that section. Note, this means that you will need to be careful when naming variables in order to avoid an infinite loop when resolving the variable substitution. Assuming that you have set an option with the name myoption in the section, you can substitute the value of that option elsewhere in the section by referring to it as %(myoption)s . Please note that the trailing s is required. Also the variable substitution will only occur when setting an option, so option names can not have a variable substitution in them.

Options and settings

If a setting is set to UNAVAILABLE or DEFAULT or left blank, osg-configure will try to use a sensible default for setting if possible.

Ignore setting

The enabled option, specifying whether a service is enabled or not, is a boolean but also accepts Ignore as a possible value. Using Ignore, results in the service associated with the section being ignored entirely (and any configuration is skipped). This differs from using False (or the %(disabled)s variable), because using False results in the service associated with the section being disabled. osg-configure will not change the configuration of the service if the enabled is set to Ignore.

This is useful, if you have a complex configuration for a given that can't be set up using the ini configuration files. You can manually configure that service by hand editing config files, manually start/stop the service and then use the Ignore setting so that osg-configure does not alter the service's configuration and status.

Configuration sections

The OSG configuration is divided into sections with each section starting with a section name in square brackets (e.g. [Section 1]). The configuration is split in multiple files and options form one section can be in more than one files.

The following sections give an overview of the options for each of the sections of the configuration files that osg-configure uses.

Site Information

The settings found in the Site Information section are described below. This section is used to give information about a resource such as resource name, site sponsors, administrators, etc. In the default installation, this section is found in /etc/osg/config.d/40-siteinfo.ini .

Option Values Accepted Description
group OSG , OSG-ITB This should be set to either OSG or OSG-ITB depending on whether your resource is in the OSG or OSG-ITB group. Most sites should specify OSG
host_name String This should be set to be hostname of the CE that is being configured
resource String The resource name of this CE endpoint as registered in OIM.
resource_group String The resource_group of this CE as registered in OIM.
sponsor String This should be set to the sponsor of the resource, if your resource has multiple sponsors, you can separate them using commas or specify the percentage using the following format 'osg, atlas, cms' or 'osg:10, atlas:45, cms:45'. The percentages must add up to 100 if multiple sponsors are used. If you have a sponsor that is not an OSG VO, you can indicate this by using local as the VO.
site_policy Url This should be a url pointing to the resource's usage policy
contact String This should be the name of the resource's admin contact
email Email address This should be the email address of the admin contact for the resource
city String This should be the city that the resource is located in
country String This should be two letter country code for the country that the resource is located in.
longitude Number This should be the longitude of the resource. It should be a number between -180 and 180.
latitude Number This should be the latitude of the resource. It should be a number between -90 and 90.

Replace at least the parts in red as appropriate:
;===================================================================
;                       Site Information
;===================================================================

[Site Information]
; The group option indicates the group that the OSG site should be listed in,
; for production sites this should be OSG, for vtb or itb testing it should be
; OSG-ITB
;
; YOU WILL NEED TO CHANGE THIS
group = OSG-ITB

; The host_name setting should give the host name of the CE  that is being
; configured, this setting must be a valid dns name that resolves
;
; YOU WILL NEED TO CHANGE THIS
host_name = ce.opensciencegrid.org

; The resource setting should be set to the same value as used in the OIM
; registration at the goc
;
; YOU WILL NEED TO CHANGE THIS
resource = GC3CE


; The resource_group setting should be set to the same value as used in the OIM
; registration at the goc
;
; YOU WILL NEED TO CHANGE THIS
resource_group = GC3

; The sponsor setting should list the sponsors for your cluster, if your cluster
; has multiple sponsors, you can separate them using commas or specify the
; percentage using the following format 'osg, atlas, cms' or
; 'osg:10, atlas:45, cms:45'
;
; YOU WILL NEED TO CHANGE THIS
sponsor = OSG

; The site_policy setting should give an url that lists your site's usage
; policy
site_policy = UNAVAILABLE

; The contact setting should give the name of the admin/technical contact
; for the cluster
;
; YOU WILL NEED TO CHANGE THIS
contact = user@opensciencegrid.org

; The email setting should give the email address for the technical contact
; for the cluster
;
; YOU WILL NEED TO CHANGE THIS
email = user@opensciencegrid.org

; The city setting should give the city that the cluster is located in
;
; YOU WILL NEED TO CHANGE THIS
city = MyTown

; The country setting should give the country that the cluster is located in
;
; YOU WILL NEED TO CHANGE THIS
country = USA

; The longitude setting should give the longitude for the cluster's location
; if you are in the US, this should be negative
; accepted values are between -180 and 180
;
; YOU WILL NEED TO CHANGE THIS
longitude = 50

; The latitude setting should give the latitude for the cluster's location
; accepted values are between -90 and 90
;
; YOU WILL NEED TO CHANGE THIS
latitude = 50

PBS

This section describes the parameters for a pbs jobmanager if it's being used in the current CE installation. If PBS is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-pbs.ini which is provided by the osg-configure-pbs RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the PBS jobmanager is being used or not.
pbs_location String This should be set to be directory where pbs is installed. osg-configure will try to loocation for the pbs binaries in pbs_location/bin.
job_contact String This should be the contact string for the pbs jobmanager (e.g. my.host.com/jobmanager-pbs)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-pbs-seg. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the PBS SEG module is enabled for the PBS job manager. Although using the SEG requires access to your PBS log files on your CE, it also substantially reduces the load on your CE and PBS scheduler since Globus will be able to monitor job status by examining the PBS logs instead of querying the PBS scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your PBS installation so that the SEG module can monitor jobs running on your cluster.
accounting_log_directory String This setting is used to tell Gratia where to find your accounting log files, and it is required for proper accounting.
pbs_server String This setting is optional and should point to your PBS server node if it is different from your OSG CE

Replace at least the parts in red as appropriate:
;===================================================================
;                              PBS
;===================================================================


[PBS]
; This section has settings for configuring your CE for a PBS job manager

; The enabled setting indicates whether you want your CE to use a PBS job 
; manager
; valid answers are True or False
enabled = FALSE

; The pbs_location setting should give the location of pbs install directory
; On rpm installations, this should be /usr if the pbs binaries (qstat, qsub, etc.) 
; are in /usr/bin
pbs_location = /usr

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-pbs) 
job_contact = host.name/jobmanager-pbs

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and determines whether the PBS SEG module is enabled
; for the PBS job manager. Although using the SEG requires access to your PBS
; log files on your CE, it also substantially reduces the load on your CE and
; PBS scheduler since Globus will be able to monitor job status by examining
; the PBS logs instead of querying the PBS scheduler.
;
; By default, this is disabled
seg_enabled = DEFAULT

; This setting is required if the SEG is enabled. This setting should point
; to the directory with your server log files for your PBS installation so
; that the SEG module can monitor jobs running on your cluster. 
;
; If you enable the SEG, you will need to change this and point it at a valid 
; directory
log_directory = UNAVAILABLE

; This setting is required for Gratia to work. It needs to read the PBS
; accounting log files in order to report accounting data. Common locations are:
; /var/torque/server_priv/accounting
; /var/lib/torque/server_priv/accounting
accounting_log_directory = UNAVAILABLE

; This setting is optional and should point to your PBS server node if it is
; different from your OSG CE 
pbs_server = UNAVAILABLE

; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

LSF

This section describes the parameters for a LSF jobmanager if it's being used in the current CE installation. If LSF is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-lsf.ini which is provided by the osg-configure-lsf RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the LSF jobmanager is being used or not.
lsf_location String This should be set to be directory where lsf is installed
job_contact String This should be the contact string for the lsf jobmanager (e.g. my.host.com/jobmanager-lsf)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-lsf. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the LSF SEG module is enabled for the PBS job manager. Although using the SEG requires access to your LSF log files on your CE, it also substantially reduces the load on your CE and LSF scheduler since Globus will be able to monitor job status by examining the LSF logs instead of querying the scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your LSF installation so that the SEG module can monitor jobs running on your cluster.

Replace at least the parts in red as appropriate:
;===================================================================
;                              LSF
;===================================================================


[LSF]
; This section has settings for configuring your CE for a LSF job manager

; The enabled setting indicates whether you want your CE to use a LSF job 
; manager
; valid answers are True or False
enabled = FALSE

; The lsf setting should give the location of the lsf install directory
lsf_location = DEFAULT

; The lsf_profile setting should give the location of the lsf_profile file
lsf_profile = DEFAULT

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-lsf) 
job_contact = host.name/jobmanager-lsf

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and determines whether the LSF SEG module is enabled
; for the LSF job manager. Although using the SEG requires access to your LSF
; log files on your CE, it also substantially reduces the load on your CE and
; LSF scheduler since Globus will be able to monitor job status by examining
; the LSF logs instead of querying the LSF scheduler.
;
; By default, this is disabled
seg_enabled = DEFAULT

; This setting is required if the SEG is enabled and for gratia accounting. This 
; setting should point to the directory with your server log files for your LSF 
; installation so that the SEG module can monitor jobs running on your cluster 
; and for the gratia probes to get your accounting information. 
;
; If you enable the SEG or gratia accounting, you will need to change this and 
; point it at a valid directory
;
log_directory = UNAVAILABLE


; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

SGE

This section describes the parameters for a SGE jobmanager if it's being used in the current CE installation. If SGE is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-sge.ini which is provided by the osg-configure-sge RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the SGE jobmanager is being used or not.
sge_root String This should be set to be directory where sge is installed (e.g. same as $SGE_ROOT variable).
sge_cell String The sge_cell setting should be set to the value of $SGE_CELL for your SGE install.
job_contact String This should be the contact string for the sge jobmanager (e.g. my.host.com/jobmanager-sge)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-sge. If it is set to False or not set then that prefix will not be added.
seg_enabled True, False This setting is optional and determines whether the SGE SEG module is enabled for the SGE job manager. Although using the SEG requires access to your SGE log files on your CE, it also substantially reduces the load on your CE and SGE scheduler since Globus will be able to monitor job status by examining the SGE logs instead of querying the SGE scheduler.
log_directory String This setting is required if the SEG is enabled. This setting should point to the directory with your server log files for your SGE installation so that the SEG module can monitor jobs running on your cluster.
default_queue String This setting determines queue that jobs should be placed in if the job description does not specify a queue.
available_queues String This setting indicates which queues are available on the cluster and should be used for validation when validate_queues is set.
validate_queues String This setting determines whether the globus jobmanager should check the job RSL and verify that any queue specified matches a queue available on the cluster. If available_queues is set, that list of queues will be used for validation, otherwise SGE will be queried for available queues

Replace at least the parts in red as appropriate:
;===================================================================
;                              SGE
;===================================================================


[SGE]
; This section has settings for configuring your CE for a SGE job manager

; The enabled setting indicates whether you want your CE to use a SGE job 
; manager
; valid answers are True or False
enabled = FALSE

; The sge_root setting should give the location of sge install directory
;
sge_root = UNAVAILABLE

; The sge_cell setting should be set to the value of $SGE_CELL for your SGE
; install.
sge_cell = UNAVAILABLE


; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-sge) 
job_contact = host.name/jobmanager-sge

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

Condor

This section describes the parameters for a Condor jobmanager if it's being used in the current CE installation. If Condor is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-condor.ini which is provided by the osg-configure-condor RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Condor jobmanager is being used or not.
condor_location String This should be set to be directory where condor is installed. If this is set to a blank variable, DEFAULT or UNAVAILABLE, the osg-configure script will try to get this from the CONDOR_LOCATION environment variable if available otherwise it will use /usr which works for the RPM installation.
condor_config String This should be set to be path where the condor_config file is located. If this is set to a blank variable, DEFAULT or UNAVAILABLE, the osg-configure script will try to get this from the CONDOR_CONFIG environment variable if available otherwise it will use /etc/condor/condor_config, the default for the RPM installation.
job_contact String This should be the contact string for the condor jobmanager (e.g. my.host.com/jobmanager-condor)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-condor. If it is set to False or not set then that prefix will not be added.

Replace at least the parts in red as appropriate:
;===================================================================
;                             Condor
;===================================================================


[Condor]
; This section has settings for configuring your CE for an HTCondor job manager

; The enabled setting indicates whether you want your CE to use an HTCondor job
; manager.
; valid answers are True or False
enabled = FALSE

; The condor_location setting should give the location of condor install directory.
; If you are using an RPM installation of HTCondor, set this to DEFAULT.
; Otherwise, set this to the root directory of the HTCondor install,
; such that $CONDOR_LOCATION/bin contains the condor binaries.
condor_location = DEFAULT

; The condor_config setting should give the location of condor config file.
; If you are using an RPM installation of HTCondor, or wish osg-configure to
; detect the correct value, set this to DEFAULT.
; On an RPM install of HTCondor, this file is located at /etc/condor/condor_config.
; On a tarball install of HTCondor, this file is typically located at
; $CONDOR_LOCATION/etc/condor_config.
condor_config = DEFAULT

; The job_contact setting should give the contact string for the jobmanager 
; on this CE (e.g. host.name/jobmanager-condor) 
job_contact = host.name/jobmanager-condor

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and will allow globus to accept limited proxies if set. 
; Changing this is not needed for almost all sites. If set to True it will add 
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork. 
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this 
; accept_limited = False

Slurm

This section describes the parameters for a Slurm jobmanager if it's being used in the current CE installation. If Slurm is not being used, the enabled setting should be set to False. This section is contained in /etc/osg/config.d/20-slurm.ini which is provided by the osg-configure-slurm RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Slurm jobmanager is being used or not.
slurm_location String This should be set to be directory where slurm is installed. osg-configure will try to location for the slurm binaries in slurm_location/bin.
job_contact String This should be the contact string for the pbs jobmanager (e.g. my.host.com/jobmanager-pbs)
util_contact String This should be the contact string for the default jobmanager (e.g. my.host.com/jobmanager)
accept_limited True, False This setting is optional and will allow globus to accept limited proxies if set. Changing this is not needed for almost all sites. If set to True it will add the "accept_limited" option into /etc/grid-services/jobmanager-pbs-seg. If it is set to False or not set then that prefix will not be added.
db_host String Hostname of the machine hosting the SLURM database. This information is needed to configure the SLURM gratia probe.
db_port String Port of where the SLURM database is listening. This information is needed to configure the SLURM gratia probe.
db_user String Username used to access the SLURM database. This information is needed to configure the SLURM gratia probe.
db_pass String The location of a file containing the password used to access the SLURM database. This information is needed to configure the SLURM gratia probe.
db_name String Name of the SLURM database. This information is needed to configure the SLURM gratia probe.
slurm_cluster String The name of the Slurm cluster

Replace at least the parts in red as appropriate:
;===================================================================
;                              SLURM
;===================================================================


[SLURM]
; This section has settings for configuring your CE for a Slurm job manager

; The enabled setting indicates whether you want your CE to use a PBS job
; manager
; valid answers are True or False
enabled = FALSE


; The slurm_location setting should give the location of slurm install directory
; On rpm installations, this should be /usr if the slurm binaries (qstat, qsub, etc.)
; are in /usr/bin
slurm_location = /usr

; The job_contact setting should give the contact string for the jobmanager
; on this CE (e.g. host.name/jobmanager-pbs)
;
; Since we're using the PBS emulation in Slurm with the globus jobamanger,
; the jobmanager should be set to pbs
job_contact = host.name/jobmanager-pbs

; The util_contact should give the contact string for the default jobmanager
; on this CE (e.g. host.name/jobmanager)
util_contact = host.name/jobmanager

; This setting is optional and will allow globus to accept limited proxies if set.
; Changing this is not needed for almost all sites. If set to True it will add
; the "accept_limited" option into /etc/grid-services/jobmanager-managed-fork.
; If it is set to False or not set then that prefix will not be added.
;
; By default this is set to False
;
; Most sites will not need to change this
; accept_limited = False


; Hostname of the machine hosting the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_host = host.name

; Port of where the SLURM database is listening.T his information is needed
; to configure the SLURM gratia probe.
db_port = port

; Username used to access the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_user = user

; Location of a file containing the password used to access the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_pass = file/location

; Name of the SLURM database. This information is needed
; to configure the SLURM gratia probe.
db_name = database_name

slurm_cluster = cluster_name


Bosco

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the Bosco jobmanager is being used or not.
users String A comma separated string. The existing usernames on the CE for which to install Bosco and allow submissions. In order to have separate usernames per VO, for example the CMS VO to have the cms username, each user must have Bosco installed. The osg-configure service will install Bosco on each of the users listed here.
endpoint String The remote cluster submission host for which Bosco will submit jobs to the scheduler. This is in the form of user@example.com, exactly as you would use to ssh into the remote cluster.
batch String The type of scheduler installed on the remote cluster.
ssh_key String The location of the ssh key, as created above.

Replace at least the parts in red as appropriate:
;===================================================================
;                              Bosco
;===================================================================

[BOSCO]
; This section has settings for configuring your CE for a BOSCO job manager

; (Required) Determines whether you want a bosco route for the compute element
enabled = TRUE

; (Required) A comma separated list of users for which to enable bosco submission
users = bosco, cms

; (Required) The endpoint should be the hostname of the remote login node, in the form:
; @
endpoint = bosco@example.com

; (Required) The type of batch system used on the remote cluster.  Possible values are:
; pbs, lsf, sge, condor, slurm
batch = slurm

; (Required) The location of the SSH key that will allow passwordless login to the
; remote cluster's login node.
ssh_key = /etc/osg/bosco.key

; (Optional) The maximum number of jobs to submit to the remote cluster, idle + running.
max_jobs = 1000

Misc Services

This section handles the configuration of services that do not have a dedicated section for their configuration. This section is contained in /etc/osg/config.d/10-misc.ini which is provided by the osg-configure-misc RPM.

This section primarily deals with authorization. For information on suggested settings for your CE, see the authorization section of the HTCondor-CE install documents.

Option Values Accepted Explanation
glexec_location String This gives the location of the glExec installation on the worker nodes, if it is present. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If it is not installed, set this to UNAVAILABLE. glExec does not work with the vomsmap authorization method on OSG 3.3 and is entirely unsupported starting in OSG 3.4
gums_host String This setting is used to indicate the hostname of the GUMS host that should be used for authentication, if the authorization method below is set to xacml. If GUMS is not used, this should be set to UNAVAILABLE. GUMS is deprecated in OSG 3.4
authorization_method gridmap, xacml, local-gridmap, vomsmap This indicates which authorization method your site uses. xacml is deprecated in OSG 3.4
edit_lcmaps_db True, False (Optional, default True) If true, osg-configure will overwrite /etc/lcmaps.db to set your authorization method. The previous version will be backed up to /etc/lcmaps.db.pre-configure
copy_host_cert_for_service_certs True, False (Optional, default False) If true, osg-configure will create a copy or copies of your host cert and key as service certs for RSV and (on OSG 3.3) GUMS

OSG 3.4 changes:

  • glexec_location must be UNAVAILABLE or unset
  • authorization_method defaults to vomsmap
  • authorization_method will raise a warning if set to xacml

RSV

This section handles the configuration and setup of the RSV services This section is contained in /etc/osg/config.d/30-rsv.ini which is provided by the osg-configure-rsv RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the rsv service is being used or not.
rsv_user String This gives username that rsv will run under. If this is blank or set to UNAVAILABLE, it will default to rsv.
gratia_probes String This settings indicates which rsv gratia probes should be used. It is a list of probes separated by a comma. Valid probes are metric, condor, pbs, lsf, sge, managedfork, hadoop-transfer, and gridftp-transfer
ce_hosts String This option lists the serviceURI of the CEs that generic RSV CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This must be set if enable_ce_probes is set. If this is left blank or set to UNAVAILABLE, the CE's hostname is used by default.
gram_ce_hosts String This option lists the serviceURI of the Globus GRAM-based CEs that the RSV GRAM CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This should be set if gram_gateway_enabled in the Gateway configuration section is True. GRAM is unsupported starting in OSG 3.4
htcondor_ce_hosts String This option lists the serviceURI of the HTCondor-CE-based CEs that the RSV HTCondor-CE probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:2812). This should be set if htcondor_gateway_enabled in the Gateway configuration section is True.
gums_hosts String This option lists the serviceURI or FQDN of the CEs or SEs, using GUMS for authentication, that the RSV GUMS probes should check. This should be a list of CE or SE FQDNs (and not a GUMS server FQDN) separated by a comma (e.g. my.host,my.host2,my.host3). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
gridftp_hosts String This option lists the serviceURI of the gridftp servers that the RSV gridftp probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host.iu.edu:2812,my.host2,my.host3). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
gridftp_dir String This should be the directory that the gridftp probes should use during testing. It must be given if enable_gridftp_probes is enabled. This defaults to /tmp if left blank or set to UNAVAILABLE.
srm_hosts String This option lists the serviceURI of the srm servers that the RSV srm probes should check. This should be a list of serviceURIs (hostname[:port/service]) separated by a comma (e.g. my.host,my.host2,my.host3:8444). This will default to the ce_hosts option if it is left blank or set to UNAVAILABLE
srm_dir String This should be the directory that the srm probes should use during testing. It must be given if enable_srm_probes is enabled.
srm_webservice_path String This option gives the webservice path that SRM probes need to use along with the host: port. For dcache installations, this should work if left blank. However Bestman-xrootd SEs normally use srm/v2/server as web service path, and so Bestman-xrootd admins will have to pass this option with the appropriate value (for example: "srm/v2/server") for the SRM probes to pass on their SE.
service_cert String This option should point to the public key file (pem) for your service certificate. If this is left blank or set to UNAVAILABLE and the user_proxy setting is set , it will default to /etc/grid-security/rsvcert.pem
service_key String This option should point to the private key file (pem) for your service certificate. If this is left blank or set to UNAVAILABLE and the service_cert setting is enabled, it will default to /etc/grid-security/rsvkey.pem .
service_proxy String This should point to the location of the rsv proxy file. If this is left blank or set to UNAVAILABLE and the use_service_cert setting is enabled, it will default to /tmp/rsvproxy.
legacy_proxy True, False Default=False. If set to true, generate a legacy Globus proxy from the service certificate using grid-proxy-init -old. This has no effect for user proxies since those are not automatically generated by RSV. New in osg-configure-1.0.8
user_proxy String If you don't use a service certificate for rsv, you will need to specify a proxy file that RSV should use in the proxy_file setting. If this is set, then service_cert, service_key, and service_proxy should be left blank, or set to UNAVAILABE or DEFAULT.
enable_gratia True, False This option will enable RSV record uploading to central RSV collector at the GOC. This should be set to True on all OSG resources (and to False on non-OSG resources).
setup_rsv_nagios True, False This option indicates whether rsv should upload results to a local nagios server instance. This should be set to True or False.
This plugin is provided as an experimental component, and admins are recommend not to enable it on production resources.
rsv_nagios_conf_file String The rsv_nagios_conf_file option indicates the location of the rsv nagios file to use for configuration details. This file needs to be configured locally for RSV-Nagios forwarding to work -- see inline comments in file for more information.
condor_location String If you installed Condor in a non-standard location (somewhere other than /usr, which is where the RPM puts it) you must specify the path to the install dir here.

OSG 3.4 changes:

  • gram_ce_hosts must be unset

Gateway

This section gives information about the options in the Gateway section of the configuration files. These options control the behavior of job gateways on the CE. Most CEs are based on HTCondor-CE, which uses condor-ce as the gateway. Some older CEs may be based on Globus GRAM, which uses globus-gatekeeper as the gateway, but GRAM is no longer supported starting in OSG 3.4.

Option Values Accepted Explanation
gram_gateway_enabled True, False (default False; unavailable starting in OSG 3.4). True if the CE is using Globus GRAM, False otherwise. GRAM Job managers will be configured for enabled batch systems. RSV will use GRAM to launch remote probes.
htcondor_gateway_enabled True, False (default True). True if the CE is using HTCondor-CE, False otherwise. HTCondor-CE will be configured to support enabled batch systems. RSV will use HTCondor-CE to launch remote probes.
job_envvar_path String The value of the PATH environment variable to put into HTCondor jobs running with HTCondor-CE. This value is ignored if not using that batch system/gateway combination.

OSG 3.4 changes:

  • gram_gateway_enabled must be False or unset

Storage

This section gives information about the options in the Storage section of the configuration file. Several of these values are constrained and need to be set in a way that is consistent with one of the OSG storage models. Please review the Storage Related Parameters section of the Environment Variables description as well as the Overview of Services? and Site Planning discussions for explanations of the various storage models and the requirements for them. This section is contained in /etc/osg/config.d/10-storage.ini which is provided by the osg-configure-ce RPM.

Option Values Accepted Explanation
se_available True, False This indicates whether there is an associated SE available. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node.
default_se String If an SE is available at your cluster, set default_se to the hostname of this SE, otherwise set default_se to UNAVAILABLE. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node.
grid_dir String The grid_dir setting should point to the directory which holds the files from the OSG worker node package. If you have installed the worker node client via RPM (the normal case) it should be /etc/osg/wn-client. If you have somehow installed the worker node in a special location (perhaps via the worker node client tarball or via OASIS), it should be the location of that directory. This directory should be visible on all of the computer nodes and will be accessed via the $OSG_GRID environment variable. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. Read access is required, though worker nodes don't need write access.
app_dir String The app_dir setting should point to the directory which contains the VO specific applications, this should be visible on both the CE and worker nodes and will be accessed via $OSG_APP. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. Only the CE needs to have write access to this directory. The $OSG_APP directory must also contain a sub-directory (etc) with 1777 permissions.
data_dir String The data_dir setting should point to a directory that can be used to store and stage data in and out of the cluster and accessed via the $OSG_DATA variable. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. This directory should be readable and writable on both the CE and worker nodes.
worker_node_temp String The worker_node_temp directory (accessed via $OSG_WN_TMP) should point to a directory that can be used as scratch space on compute nodes. It should allow read and write access on a worker node and can be visible to just that worker node. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE
site_read String The site_read setting should be the location or url to a directory that can be read to stage in data via the variable $OSG_SITE_READ. This is an url if you are using a SE. Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE
site_write String The site_write setting should be the location or url to a directory that can be write to stage out data via the variable $OSG_SITE_WRITE. This is an url if you are using a SE . Can be defined in terms of an environment variable (e.g. $FOO) that will be evaluated on the worker node. If not set, the default is UNAVAILABLE

Replace at least the parts in red as appropriate:
;===================================================================
;                            Storage 
;===================================================================

[Storage]
;
; Several of these values are constrained and need to be set in a way
; that is consistent with one of the OSG storage models
;
; Please refer to the OSG release documentation for an indepth explanation 
; of the various storage models and the requirements for them

; If you have a SE available for your cluster and wish to make it available 
; to incoming jobs, set se_available to True, otherwise set it to False
se_available = FALSE

; If you indicated that you have an se available at your cluster, set default_se to
; the hostname of this SE, otherwise set default_se to UNAVAILABLE
default_se = UNAVAILABLE

; The grid_dir setting should point to the directory which holds the files 
; from the OSG worker node package, it should be visible on all of the computer
; nodes (read access is required, worker nodes don't need to be able to write) 
; 
; For RPM installations, /etc/osg/wn-client is the correct location
grid_dir = /etc/osg/wn-client/

; The app_dir setting should point to the directory which contains the VO 
; specific applications, this should be visible on both the CE and worker nodes
; but only the CE needs to have write access to this directory
; 
; YOU WILL NEED TO CHANGE THIS
app_dir = /osg/app

; The data_dir setting should point to a directory that can be used to store 
; and stage data in and out of the cluster.  This directory should be readable
; and writable on both the CE and worker nodes
; 
; YOU WILL NEED TO CHANGE THIS
data_dir = UNAVAILABLE

; The worker_node_temp directory should point to a directory that can be used 
; as scratch space on compute nodes, it should allow read and write access on the 
; worker nodes but can be local to each worker node
; 
; YOU WILL NEED TO CHANGE THIS
worker_node_temp = /scratch

; The site_read setting should be the location or url to a directory that can 
; be read to stage in data, this is an url if you are using a SE 
; 
; YOU WILL NEED TO CHANGE THIS
site_read = UNAVAILABLE

; The site_write setting should be the location or url to a directory that can 
; be write to stage out data, this is an url if you are using a SE 
; 
; YOU WILL NEED TO CHANGE THIS
site_write = UNAVAILABLE

Squid

This section handles the configuration and setup of the squid web caching and proxy service. This section is contained in /etc/osg/config.d/01-squid.ini which is provided by the osg-configure-squid RPM.

Option Values Accepted Explanation
enabled True, False, Ignore This indicates whether the squid service is being used or not.
location String This should be set to the hostname:port of the squid server.

Replace at least the parts in red as appropriate:
;===================================================================
;                             Squid
;===================================================================

[Squid]
; Set the enabled setting to True if you have squid installed and wish to 
; use it, otherwise set it to False 
enabled = FALSE

; If you are using squid, specify the location of the squid server in the 
; location setting, this should be an url
location = DEFAULT

; If you are using squid, use the policy setting to indicate which cache
; replacement policy squid is using
policy = DEFAULT

; If you are using squid, use the cache_size setting to indicate which the 
; size of the disk cache that squid is using
cache_size = DEFAULT

; If you are using squid, use the memory_size setting to indicate which the 
; size of the memory cache that squid is using
memory_size = DEFAULT

Subcluster / Resource Entry Configuration

Subcluster and Resource Entry configuration is for reporting about the worker resources on your site. A subcluster is a homogeneous set of worker node hardware; a resource is a set of subcluster(s) with common capabilities that will be reported to the ATLAS AGIS system.

At least one Subcluster or Resource Entry section is required on a CE; please populate the information for all your subclusters. This information will be reported to a central collector and will be used to send GlideIns / pilot jobs to your site; having accurate information is necessary for OSG jobs to effectively use your resources.

For historical reasons, this configuration is specified in the file named 30-gip.ini, installed by the osg-configure-gip RPM.

This configuration uses multiple sections of the OSG configuration files:

Notes for multi-CE sites.

If you would like to properly advertise multiple CEs per cluster, make sure that you:
  • Set the value of site_name in the "Site Information" section to be the same for each CE.
  • Have the exact same configuration values for the Subcluster* and Resource Entry* sections in each CE.

Subcluster Configuration

Each homogeneous set of worker node hardware is called a subcluster. For each subcluster in your cluster, fill in the information about the worker node hardware by creating a new Subcluster section with a unique name in the following format: [Subcluster CHANGEME], where CHANGEME is the globally unique subcluster name (yes, it must be a globally unique name for the whole grid, not just unique to your site. Get creative.)

Option Values Accepted Explanation
name String The same name that is in the Section label; it should be globally unique
ram_mb Positive Integer Megabytes of RAM per node
cores_per_node Positive Integer Number of cores per node
allowed_vos Comma-separated List or * The VOs that are allowed to run jobs on this subcluster (autodetected if *). Optional on OSG 3.3

The following attributes are optional:

max_wall_time Positive Integer Maximum wall-clock time, in minutes, that a job is allowed to run on this subcluster (the default is one day or 1440 mins)
queue String The queue to which jobs should be submitted in order to run on this subcluster
extra_transforms Classad Transformation attributes which the HTCondor Job Router should apply to incoming jobs so they can run on this subcluster

OSG 3.4 changes:

  • allowed_vos is mandatory

Resource Entry Configuration (ATLAS only)

If you are configuring a CE for the ATLAS VO, you must provide hardware information to advertise the queues that are available to AGIS. For each queue, create a new Resource Entry section with a unique name in the following format: [Resource Entry RESOURCE] where RESOURCE is a globally unique resource name (it must be a globally unique name for the whole grid, not just unique to your site). The following options are required for the Resource Entry section and are used to generate the data required by AGIS:

Option Values Accepted Explanation
name String The same name that is in the Resource Entry label; it must be globally unique
max_wall_time Positive Integer Maximum wall-clock time, in minutes, that a job is allowed to run on this resource
queue String The queue to which jobs should be submitted to run on this resource
cpucount (alias cores_per_node) Positive Integer Number of cores that a job using this resource can get
maxmemory (alias ram_mb) Positive Integer Maximum amount of memory (in MB) that a job using this resource can get
allowed_vos Comma-separated List or * The VOs that are allowed to run jobs on this resource (autodetected if *). Optional on OSG 3.3

The following attributes are optional:

Option Values Accepted Explanation
subclusters Comma-separated List The physical subclusters the resource entry refers to; must be defined as Subcluster sections elsewhere in the file
vo_tag String An arbitrary label that is added to jobs routed through this resource

OSG 3.4 changes:

  • allowed_vos is mandatory

Local Settings

This section differs from other sections in that there are no set options in this section. Rather, the options set in this section will be placed in the osg-local-job-environment.conf verbatim. The options in this section are case sensitive and the case will be preserved when they are converted to environment variables. The osg-local-job-environment.conf file gets sourced by jobs run on your cluster so any variables set in this section will appear in the environment of jobs run on your system.

Adding a line such as My_Setting = my_Value would result in the an environment variable called My_Setting set to my_Value in the job's environment. my_Value can also be defined in terms of an environment variable (i.e My_Setting = $my_Value) that will be evaluated on the worker node. For example, to add a variable MY_PATH set to /usr/local/myapp, you'd have the following:

[Local Settings]

MY_PATH = /usr/local/myapp

This section is contained in /etc/osg/config.d/40-localsettings.ini which is provided by the osg-configure-ce RPM.

Replace at least the parts in red as appropriate:
;
; Put any environment variables that should go into the local
; job environment for grid jobs here.  The format is
;   env = setting
; and case is preserved, the setting does not need to be quoted
; See https://twiki.grid.iu.edu/bin/view/ReleaseDocumentation/ConfigurationFileHelp#Local_Settings
; for more details
[Local Settings]

Info Services

Reporting to the central CE Collectors is configured in this section. For historical reasons, this is found in 30-infoservices.ini, installed by the osg-configure-infoservices RPM. In the majority of cases, this file can be left untouched; you only need to configure this section if you wish to report to your own CE Collector instead of the ones run by OSG Operations.

Option Values Accepted Explanation
enabled True, False, Ignore True if reporting should be configured and enabled.
ce_collectors String The server(s) HTCondor-CE information should be sent to. Setting this to DEFAULT will report to the OSG Production or ITB servers (depending on your Site Information configuration);
setting this to PRODUCTION will report to the OSG Production servers;
setting this to ITB will report to the OSG ITB servers;
otherwise, set to the hostname:port of a host running a condor-ce-collector daemon.

Replace at least the parts in red as appropriate:
;===================================================================
;                         Info Services
;===================================================================
;
[Info Services]
; This setting indicates which servers HTCondor-CE information
; should be sent to. These servers should be running a
; condor-ce-collector daemon. Set this to a comma-separated
; list of hostname:port combinations. The default collector port
; 9619 will be used if not specified. You can also use one of these
; special values:
; 'PRODUCTION' : OSG production servers
; 'ITB'        : OSG testing servers
; 'DEFAULT'    : OSG testing servers if this is an ITB site,
;                production otherwise
ce_collectors = DEFAULT

Gratia

This section configures Gratia. If probes is set to UNAVAILABLE, then osg-configure will use appropriate default values. If you need to specify custom reporting (e.g. a local gratia collector) in addition to the default probes, %(osg-jobmanager-gratia)s, %(osg-gridftp-gratia)s, %(osg-metric-gratia)s, %(itb-jobmanager-gratia)s, %(itb-gridftp-gratia)s, %(itb-metric-gratia)s are defined in the default configuration files to make it easier to specify the standard osg reporting. This section is contained in /etc/osg/config.d/30-gratia.ini which is provided by the osg-configure-gratia RPM.

Option Values Accepted Explanation
enabled True , False, Ignore This should be set to True if gratia should be configured and enabled on the installation being configured.
resource String This should be set to the resource name as given in the OIM registration
probes String This should be set to the gratia probes that should be enabled. A probe is specified by using as [probe_type]:server:port . Valid probe_types are metric (for rsv), jobmanager (for the appropriate jobmanager probe) and gridftp for the gridftp-transfer-probe.

Replace at least the parts in red as appropriate:
;===================================================================
;                         Gratia
;===================================================================

[Gratia]

; Default gratia servers
;
; Please don't change these unless you have good reason to do so

; Variables for osg itb probes, CE installations should use the
; jobmanager probe
itb-jobmanager-gratia = jobmanager:gratia-osg-itb.opensciencegrid.org:80
itb-gridftp-gratia = gridftp:gratia-osg-itb.opensciencegrid.org:80


; Variables for osg production probes, CE installations should use the
; jobmanager probe
osg-jobmanager-gratia = jobmanager:gratia-osg-prod.opensciencegrid.org:80
osg-gridftp-gratia = gridftp:gratia-osg-transfer.opensciencegrid.org:80


; The enable option indicates whether gratia should be enabled or
; disabled.  It should be set to True or False
;
; You generally want Gratia enabled for any CE installation, in
; addition SE installations may want to enable gratia to use
; the gratia gridftp reporting
enabled = TRUE


; This setting specifies the resource that gratia will use to report
; accounting information, on a CE if you leave this blank, gratia will
; use the resource setting from the Site Information section
;
resource = DEFAULT

; This setting indicates which probes should be enabled for gratia
; The list should be given as probe_name1:host1:port1, probe_name2:host2:port2
; where probe_name is either jobmanager or gridftp
; host is a fully qualified domain name
; port is the port that the server is listening on
; CEs should have entries for jobmanager probe
; SEs should use gridftp if they would like to enable gridftp transfer
; accounting
;
; for convenience admins can use %(osg-jobmanager-gratia)s, and
; %(osg-gridftp-gratia)s for production sites
; and %(itb-jobmanager-gratia)s, and %(itb-gridftp-gratia)s for ITB sites
probes = %(itb-jobmanager-gratia)s,%(itb-gridftp-gratia)s

Comments

Topic revision: r47 - 06 Jun 2017 - 23:57:29 - MatyasSelmeci
Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..