Local Storage Configuration

Introduction to local storage configuration conventions

This document describes how to configure the OSG attributes, including ones referencing "CE storage", during the installation and afterwards (if the CE layout needs to change).

Configuring local storage attributes

You must make these attributes known to OSG. They will be published as part of the GLUE schema using the GIP and used directly or indirectly by other OSG applications and users submitting jobs. The configuration is set in $VDT_LOCATION/monitoring/osg-attributes.conf. This is a standard configuration file that you can edit or review directly. The configuration script $VDT_LOCATION/monitoring/configure-osg.py automates much of the configuration. The meaning and purpose of the various elements of the configuration attributes are documented further below, and in the GLUE documentation. New resource administrators may want to read that information carefully and determine how to map those elements onto their Resource before proceeding. Guidance on the basic elements and common defaults is provided below.

Gather configuration information

OSG strives to make resources available with minimal requirements; however, the grid requires certain information about files and filesystem mount points to provide a basic execution environment. For applications to be installed and to be executed correctly, filesystem sharing and the filesystem mount points available for a cluster must be specifically coordinated. For this purpose, administrators must define special directory hierarchies (mount points) and allocate them in the OSG environment. Many of these mount points should be available on the head / gatekeeper node and, using the exact path, on each of the worker nodes. Generally, they do not have to be made available in the form of a shared filesystem across the whole cluster. Read-only spaces can generally be provisioned with or without a shared filesystem as long as you provide consistent paths.


This points to the OSG software installation location on the CE. It must be writable by root. This attribute is automatically setup by the configure-osg.py script. The $OSG_LOCATION directory should not be exported to the worker nodes.


Where OSG worker-node client software is installed; see Worker Node Client for a description. $OSG_GRID includes client utilities for Grid middleware, such as srmcp. It should be writable by root and readable by all users. It must be accessible by both gatekeeper and worker nodes via a shared filesystem, or different installations on local disks using a consistent pathname.


Base location for VO-specific application software.

$OSG_APP is read-only mounted on all worker nodes in the cluster. Only users with software installation privileges in their VO should have write privileges to this directories. At least 10 GB of space should be allocated per VO.

The $OSG_APP directory also requires an ./etc sub-directory with 1777 permissions.


The data directories are intended as the spaces for applications to write input and output data files with persistency that must exceed the lifetime of the job which created it.
  • These directories should be writable by all users.
  • Users will be able to create sub-directories which are private, as provided by the filesystem.
  • At least 10 GB of space should be allocated per worker node; some VOs require much larger allocations.

The following different options are possible:

  • $OSG_DATA: shared directory with read-write access for all users
  • $OSG_SITE_READ: shared directory with read-only access for all users (data may be prestaged by the administrator or using a SE pointing to the same space)
  • $OSG_SITE_WRITE: shared directory with write-only access for all users (data may be staged out by the administrator or using a SE pointing to the same space)

A CE can provide $OSG_DATA, both $OSG_SITE_READ and $OSG_SITE_WRITE, or none of them if it has a local SE specified in $OSG_DEFAULT_SE. If a particular hierarchy is not available on your CE, provide the keyword UNAVAILABLE.

The $OSG_DATA, $OSG_SITE_READ and $OSG_SITE_WRITE directories must be accessible from the head node as well as each of the worker nodes.


A temporary directory local to the worker node, used as a working directory.
  • At least 10 GB per virtual CPU should be available in this directory (e.g. a WorkerNode with 2 hyperthreaded CPUs that can run up to 4 jobs, should have 40GB).
  • Files placed in this area by a job may be deleted upon completion of the job.


A storage element that is close and visible from all the nodes of the CE, both worker and head node.

The value to be specfied in $OSG_DEFAULT_SE is the full URL, including method, host/port and path of the base dir. This full URL must be reachable from inside as well as outside the cluster. The $OSG_DEFAULT_SE generally supports only put and get, rather than open/read/write/close.

If the CE has no default SE it can use the value UNAVAILABLE for $OSG_DEFAULT_SE.

The current release supports SRM and gftp for $OSG_DEFAULT_SE.

Execute the configuration script

Run the following script as root to execute the configuration script.

> cd VDT_LOCATION/monitoring
> cd $VDT_LOCATION/monitoring
> ./configure-osg.py

The configure-osg.py script creates the $VDT_LOCATION/monitoring/osg-attributes.conf file and grid3-info.conf that is a link to it. The script also creates $VDT_LOCATION/monitoring/osg-job-environment.conf which duplicates some of the attributes from $VDT_LOCATION/monitoring/osg-attributes.conf so that they can be placed in the job environment as well as $VDT_LOCATION/monitoring/osg-local-job-environment.conf. The $VDT_LOCATION/monitoring/osg-local-job-environment.conf file is to allow site admins to set site specific environment variables that should be present for jobs that are running on the cluster. The This file is used by several monitoring services and applications to obtain basic resource configuration information. This file is required to be readable from that location for OSG CEs.

The resource owner may choose which information services to run to advertise this information. Configuration of several of the more popular ones is described below in the Monitoring section.

Reconfiguring an Existing OSG Site

An already installed OSG site can be reconfigured by editing osg-attributes.conf, or by rerunning the configure-osg script.

Storage Types Supported at OSG Sites

All OSG sites should support the following storage:

Name Purpose Notes
$OSG_APP Install application software Exported to compute nodes
$OSG_DATA Stage output files from worker nodes, for later retrieval using Gridftp  
$OSG_GRID Set of "client tools" that are part of the OSG software stack. Available from the compute node
$OSG_WN_TMP1 Temporary storage area in which your job runs Local to each worker node

  1. The standard is to start your job(s) in a dedicated directory on one of the worker node's local disks and configure $OSG_WN_TMP to point to that directory. However, many OSG sites simply place your jobs on some shared filesystem and expect you to do the following before starting the job:
    export mydir=$OSG_WN_TMP/myWorkDir$RANDOM ; mkdir $mydir ; cd $mydir
    . This is particularly important if your job has significant I/O.

Input and Output Files Specified via Condor-G JDL

Condor-G allows specification of the following:

executable one file
output one file - stdout
error one file - stderr
transfer_input_Files 1 comma seperated list of files
transfer_output_files 2 comma seperated list of files

Do not use to transfer more than a few megabytes: these files are transfered via the CE headnode and can cause serious loads, which can bring down the cluster.

Do not spool gigabyte-sized files via the CE headnode by condor file transfer. Space on the headnode tends to be limited, and some sites severely quota the gram scratch area via which these files are spooled. Instead, store them in the dedicated stage-out spaces and pull them from the outside as part of a DAG.

In the remainder of this NOTE% document we describe how to find your way around these various storage areas both from outside the site before you submit your job, as well as from inside the site after your job starts inside a batch slot.

Finding your way around

A definition of the concepts used in this document can be found in Storage Models. This page also indicates which elements each storage model requires.

Here we describe how to find the various storage locations. We start by describing how to find things after your job starts, and then complete the discussion by describing what you can determine from the outside, before you submit a job to the site. We deliberately do not describe what these various storage implementations are.

Requirements for Configuring Local Storage

This topic describes how site administrators can define CE local storage paths so that their site will be configured in a way that OSG users expect.

Local storage definitions define the paths to the disk spaces or Storage Elements (SEs) that are accessible to jobs from within a OSG Compute Element (CE). These can be set in a variety of ways depending on what you need for your CE and your site configuration.

This section also covers how these environment variables correspond to external schemas (e.g. GLUE, Grid3/OSG).

Provide the keyword UNAVAILABLE (instead of a path) if your CE does not support a particular CE storage area. This distinguishes CEs that provide support for only certain CE storages from those that simply are not configured.

These values do not refer to any storage space as viewed from outside (e.g., through a Storage Element or GridFTP server).

OSG - LDAP - Glue table

The following table shows the OSG storage variable name, its associated attribute name in the GLUE Schema 1.2, its LDAP attribute name (as in Grid3), and a description.

The same attribute may appear in more than one place in the GLUE Schema.

OSG variable GLUE attribute LDAP attribute Description
$OSG_GRID Location.Path (2) Gri3Dir? Read-only area for OSG worker node's client software
$OSG_APP CE.Info.ApplicationDir (CE.Info.ApplicationDir) (1) Gri3AppDir? VO-wide software installations
$OSG_DATA CE.Info.DataDir (CE.VOView.DataDir) (1) Gri3DataDir? Transient storage shared between jobs executing on the worker nodes
$OSG_SITE_READ Location.Path (2)   Transient storage visible from all worker nodes and optimized for high-performance read operations
$OSG_SITE_WRITE Location.Path (2)   Transient storage visible from all worker nodes and optimized for high-performance write operations
$OSG_WN_TMP CE.Cluster.WNTmpDir (CE.SubCluster.WNTmpDir)   Temporary work area that may be purged when the job completes
$OSG_DEFAULT_SE CE.Info.DefaultSE (CE.VOView.DefaultSE)   Storage Element closely related to the CE, only accessible using SE access methods

  1. GLUE provides the possibility to have multiple values for some of the CE storage, depending on the VO and the Role (VOMS FQAN). These are currently sitewide information within OSG.
  2. The GLUE Schema does not have an specific attribute for SITE_WRITE or SITE_READ, but it provides the location entity (Name/Version/Path sets) to accommodate additional CE local storage. In order to accommodate that, locations will have to be defined through the GIP:
    1. LocalID: GRID+OSG, Name:GRID, Version: OSG, Path:
    2. LocalID: SITE_WRITE+OSG, Name:SITE_WRITE, Version: OSG, Path:
    3. LocalID: SITE_READ+OSG, Name:SITE_READ, Version: OSG, Path:

Local Storage Configuration Models

Although you are not required to provide all of the OSG storage area definitions, you must implement at least one of the OSG storage models. Providing a wider selection of CE storage areas allows users to select the one that best fits their jobs. It can also allow users to define jobs with inefficient execution models that could reduce the performance of the whole cluster. It's a trade-off.

However, unless there is a very good reason not to, you should provide $OSG_DATA for users since many VOs depend on this being present for their applications to use.

Methods of Deploying CE Storage Definitions

Declaring a storage area to be local to the CE can be done in a variety of ways, including (but not limited to):

  • Defining environment variables that resolve to the correct path or URL for the storage areas.
  • Making paths or URLs consistent across the CE (headnodes and WNs), and publishing this using an information provider (e.g. GIP/BDII). Users would have to do a lookup before submitting the job, so that it can carry the information necessary to use this CE.

Each CE administrator must ensure the client software will function correctly (i.e., for Globus and SRM, access to the user's proxy, gass_cache mechanism) for all jobs running on the CE. Common practice is to use a shared $HOME directory, but you can use other mechanisms as long as they are transparent to the users, who should not have any assumptions about the CE that differ from what is stated here. It is unsafe to make assumptions about the existence and characteristics (size, being shared, etc.) of the $HOME directory. In particular, site administrators are free to deploy configurations that do not include any NFS exports from the CE, such as those described in OSG document 382.

Complete: 2
Responsible: SuchandraThapa - 18 Oct 2007
Reviewer - date: RobGardner - 08 Nov 2007
Comment: Needs some editing for clarity
Comment: JohnWeigand - 28 Jan 2009 - updated the $OSG_APP section for the ./etc sub-directory and permissions
Review status: NEW

Topic revision: r39 - 10 Oct 2011 - 21:22:19 - JamesWeichel

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..