The Open Science Grid provides a BDII service which aggregates LDAP information provided by sites. The information describes the resources being made available by the sites. For example, some site configuration parameters may be published from which it can be determined whether an application submitted by a VO may successfully run. In some applications, knowledge of site configuration values is needed in order to correctly use a resource. As an example, in the use of storage, a site may offer a Storage Element for which particular VOs are authorized. Each authorized VO would be able to access the Storage Element though a protocol published in the BDII, directing data to and from a storage area assigned to them.
The OSG Discovery Tool allows users to discover the BDII information they need in order to use grid resources. The version of the tool described here is focussed on storage applications.
More detailed information on the requirements and design of the tool can be found at
OSG Discovery Tool Requirements Document
OSG Discovery Tool Design Document
The version of the search tool that is described in these pages can be downloaded from the Search Tool Build Repository
. The package will be found there as a tarred, gzipped file with checksum information. To install the tool, simply untar the package in a directory of your choice.
Running the Scripts
There are some optional environment variables which may be set on the
command line or in the conf/xpathsearch.rc
- The URI of the LDAP server and the base to be searched from may be set with LDAP_URI and LDAP_BASE, respectively. These default to the OSG production BDII server and "o=grid" base.
- The cache location (see below) may be set with XPATHCACHE_DIR. The expiration time of the cache is set with MAX_CACHE_AGE. The default value is 86400 seconds.
- Commands will run slightly faster if the JAVA_HOME variable is set.
- For convenience, you may wish to add the "bin" directory of the installation to your PATH variable, however, the scripts will also work when called using full or relative paths.
Loading the Cache
The cache is loaded automatically when a query is run and the cache file is not present or is older than the value of MAX_CACHE_AGE. To load the cache manually, type
This gathers all needed information from the BDII. Queries are evaluated against the cache (except for xpathsearh, see Advanced Usage below). The above command replaces any exsisting cache with a new one. By default, the cache location is the "cache" directory of the installation. This may be changed by setting XPATHCACHE_DIR in the xpathsearch.rc file.
Java version 1.5 or higher is required. The scripts will search for a java binary in the expected locations, starting with JAVA_HOME, until one of sufficient version is found. Therefore, if you wish to specify the java binary to be used, set the JAVA_HOME environment variable.
General Command Line Format
Each script is called with a set of parameters specified on the command line, some of which may be optional. The command line format is as follows
script_name --parameter_name1 value1 --parameter_name2 value2 ...
All scripts use this form of name/value pair command line arguments. The help argument is the form of "-h", "-help", or "--help", is supported by all scripts and must be the first argument when used. When the help argument is specified, a script will print it's purpose and the list of parameters that it takes, with descriptions and indications of which are optional.
The Search Scripts with Examples
Most of the scripts in the current tool release make queries about Storage Elements. Each script searches the cached information loaded from the BDII and composes a command for each site that supports the VO specified. The command for a particular site can then be run by the user to make a space reservation or to do a test transfer. The use of each script supplied in the bundle is described in the following topic, with examples provided.
Opportunistic Storage Search Scripts?
For further information on the specific space reservation and srm copy commands supported, see the OSG Storage Documentation
How the Scripts Work
The scripts work by evaluating xqueries on the BDII data which has been transformed to XML. The evaluation is done using two components. The caching component extracts the paths from all xquery scripts and converts them to LDAP searches. The search is done on the OSG BDII endpoint, and the results are converted into XML. The XML output is saved in the "cache" directory as a file. All is done using the Glue Schema, version 1.3, employed by the BDII service. In the second step, the xquery script is evaluated on the resulting XML file using the Saxon processor.
The XQuery Scripts
Information on the xquery scripts used by the tool may be found at
XQuery scripts for composition of commands
Searching the BDII using XPath Expressions
You may search the BDII directly using XPath expressions written using Glue Schema 1.3. Please see the following topic for a description.
Running XQuery Scripts
If you have written your own xquery script, you may run it against the OSG BDII using the included "xquerybase" utility like so
xquerybase my_xquery_script.xq parameter_name1=value1 parameter_name2=value2 ...
The parameters are external variables defined in your script.
- 08 Dec 2009