Components of a Tier 3: Installation and Configuration
Use this document and the referred ones as a catalog.
You can check here why you may be interested in a component or you may jump directly to the page of the component that you want to install or configure.
All the modules in this section can be installed separately or grouped together on different nodes of the cluster. In general, there are four phases to the installation process although not all components of each section will be necessary:
- Hardware, OS setup, and networking
- Installation of Condor (or other batch scheduler)
- Storage Element and/or Compute Element installation
- Experiment specific software (see the external documentation provided by the experiment)
- Fig. 1 - Stack of the different available modules presented: phase 1 is in red, phase 2 in yellow, phase 3 in blue. Lower modules are functional to the modules sitting on top or inside them. Included boxes enhance/modify the container. Dashed boxes are optional (if used affect the components on top of them, but are not required). Click here for an alternative view.
Phase 1 -- Hardware, OS setup, and networking
To ease the following installation steps and to have a common base here are established some conventions and customizations of the base OS, like user accounts, a shared file system and others.
Basic operating system
Each node must have its own operating system (OS). OS installation is not covered by this document. For convenience we have added the ClusterOSInstall
document that contains links to experiment specific documentation describing the OS installation process.
The reference OS used in this document is Scientific Linux 5.4, a linux distribution based on RHEL 5.4. The instructions and examples in this document should work for any RHEL based OS (e.g. RHEL, SL variants, CentOS, ...) with little or no variation.
Network and time configuration
Nodes are connected according to one of the topologies described in the section about the cluster network
. Read further the network setup
section for a checklist and tips on the network configuration.
Time synchronization is essential to Grid services and other secure services. Without a correct time and date you'll incur in authentication errors. NTP is a service that synchronizes hosts using the network. Probably the OS installation will take care of it. ClusterTimeSetup
explains how to solve synchronization problems and how to modify your NTP configuration.
File system structure naming conventions (including NFS conventions)
Here is part of the standard file system structure in a Tier 3:
/exports - root of the NFSv4 pseudo filesystem (directory tree exported by NFS)
/nfs - collects the directories mounted from NFS
/home - home directories (shared)
/opt - grid software
/scratch - local scratch area
/storage - data disk used for distributed file system (xrootd)
Read more about the file system structure in ClusterFileSystem
A first option, suitable for small Tier 3s makes use of shared directories to ease administration. ClusterNFSSetup
describes how to setup the server and the clients for NFSv4
A second option performs the installation without the use of shared disk space.
If you decide not to have any NFS file system you will be limited in the possible options (e.g. no shared Condor installation) and some of the administrative tasks will be more difficult.
Security: Setting up SSH
To make the cluster secure we'll use a host-based SSH key infrastructure. This section will introduce keys, agents, how to configure them and how to use them to ease in a safe way the administration of the Tier 3.
describes an example of SSH configuration and use. That is a suggested setup, make sure to follow the policies and recommendation in place at your university or laboratory.
Setting up user and system accounts
describes the configuration of the user accounts
Phase 2 -- Quick guide for setting up a Condor cluster
Each cluster needs a Local Resource Manager
(LRM, also called queue manager
) to schedule the jobs on the worker nodes. Common LRMs are Condor, LSF, PBS, SGE; if you have one installed or a preferred one you can use that one.
If not, this section describes how to install and configure Condor 7.4, the latest production version of Condor.
We recommend to install Condor using the RPM packages provided by the Condor team:
- CondorRPMInstall - Condor is installed on each node using the RPM packages of Condor
- no contention for shared files
- only option if no shared file system
- cons: some more work to add nodes
- cons: updates need to be installed on all nodes
Alternatively you may install Condor using the TAR file distribution on a shared file system. This may become necessary if you have a platform not supported by the current RPMs or if you need to have multiple version of Condor installed and selectable at the same time:
- CondorSharedInstall - an easy way to add Condor to small clusters using a shared file system
- supports more platforms (not only RHEL derived ones)
- allows the concurrent suport of multiple versions
- it is easy to add new nodes
- configuration changes and updates are also very quick
- cons: shared installation may slow down several nodes using it
The base configuration of condor can be modified to add some extra features:
- CondorServiceJobSlotSetup describe how to add a service jobs slot to each node. This will allow to install application software or to run monitoring jobs on the worker nodes without modify the scheduling of the regular jobs
- CondorHawkeyeSetup describes how to setup Condor Startd Cron (a.k.a. Hawkeye) and shows an example using Hawkeye to conditionally schedule or suspend jobs depending on the activity on the worker node
show how to check the status of the Condor installation and how to submit some simple job.
Phase 3 -- User Interface, Compute Element (CE), Storage Element (SE)
Setting up the Pacman Package Management System
Pacman is a package management program used to install most of OSG software.
describes how to install Pacman.
as the installation directory.
tar --no-same-owner -xzvf pacman-3.29.tar.gz
ln -s pacman-3.29 pacman
Once Pacman is installed you can
and you are ready to install OSG software using Pacman.
User client (Interactive nodes)
The user client machine is an interactive node that allows users to login and provides them with software to access Grid services and specific VO software. This document covers only the installation of Grid clients, leaving to the VOs to document specific application software installation.
shows the installation of the Grid client software.
Note that users will need to request a certificate
and register in a VO
before being able to use the Grid.
Examples from VOs:
Grid user authentication
There are 2 options:
- Use a gridmap file
- Use a GUMS server
Setting up an optional Xrootd Distributed File System
xrootd is a shared file system distributed across several nodes, each node serving a portion of the file system. Xrootd is optimized to store data files; it is not for programs or system files. Maximum efficiency is obtained when jobs are running on the node that serves the input files locally.
You may want to install xrootd if you have large ROOT data files and would like an uniform file system space using the disk space available on several nodes (dedicated data servers or the worker nodes).
describes the installation of the xrootd file system.
Worker node client
The worker node client is used to allow non interactive jobs (the ones submitted using the Grid or Condor) to access the Grid.
We are installing the worker node client in
, linked to
using the instruction in the OSG release documents: WorkerNodeClient
On the node chosen for the installation, e.g.
, you can use the following:
ln -s /nfs/osg/wn /osg/wn
pacman -get http://software.grid.iu.edu/osg-1.2:wn-client
ln -s /etc/grid-security/certificates /osg/wn/globus/TRUSTED_CA
On all other nodes that may run jobs, e.g. the worker nodes:
ln -s /nfs/osg /osg
Alternatively, if there is no shared file system or if there is a load concern, the worker node client can be installed locally, in
on all the worker nodes and
will point to this directory (
ln -s /opt/wn /osg/wn
Installing a Compute Element
The installation of the OSG CE is described in the Compute Element installation guide from the release documentation
. A simplified version is being written in ComputeElementInstall
Installing a Storage Element (SE)
All T3 sites incorporate a Storage Element for remotely accessing, managing, and moving large datasets. Storage elements can be broken down into three logical components although structurally these are often packaged together in different combinations for easier installation. The three components are:
- Storage Resource Manager (SRM) - An SRM is a standardized interface to a GridFTP server and is the only way to uniform way to guarantee write access on OSG resources; it is essential for sites that share storage resources on the Grid. Additionally, CMS management recommends that all CMS T3 sites deploy an SRM. Technically this is an optional component since a stand alone GridFTP server can serve as a gateway.
- The advantages are:
- It is no more difficult to deploy than a stand-alone GridFTP server.
- It provides standardized grid commands for users.
- All Tier 2s deploy an SRM so there is ample support from the Tier 2 community should problems arise.
- The only disadvantage is that an SRM adds an additional layer of software to the stack. This can sometimes obscure problems when debugging. Note: the Atlas experiment is considering not making SRMs a requirement for their Tier 3s but this has not been decided as of 12/09.
- A GridFTP server - GridFTP is the standard way for moving large datasets across the grid, and is required for data subscription models. Since GridFTP servers are integrated with file systems, there can be multiple servers per site although this is not typical for Tier 3 systems and is not described here.
- One or more file systems - File systems can be as simple as an attached disk array, but it is recommended that Tier 3 sites setup a shared NFS file system that is accessible by all the nodes as well as GridFTP. Additionally, depending upon the analysis application requirements, some sites (especially Atlas) will want to set up a separate distributed file system such as Xrootd.
The OSG BeStMan-gateway installation provides an SRM interface coupled with a GridFTP server and is the recommended path for most Tier 3s. To setup a BeStMan-gateway follow the BeStManGateway installation guide
. If you need to install a standalone GridFTP server or an additional GridFTP server for your BeStMan installation , please, follow this Installation guide
If you need to install a stand alone Xrootd file system please refer to the Xrootd file installation guide section above.
If you have installed an Xrootd file system and need to enable authenticated and authorized file transfers on top of your Xrootd installation , please, follow this Installation guide
Additional links to documentation:
Squid web proxy
A Squid Web proxy may be needed as a best practice recommendation for larger systems (> 10 machines) to keep the CRLs updated. Alternatively, there may be a local server that caches the CRLs for the local sites.
NDT, OWAMP, BWCTL and NPAD are a set of tools provided by Internet2 and packaged and distributed by VDT. They interact with the Network Performance Toolkit that has been installed on dedicated machines on OSG Sites.
Network Performance Toolkit
describes the tools and provides instructions for their installation and use.
Once your site is ready with all the desired services you should register it in OIM
to add the site, resource, officially in OSG. The OIM registration is important because:
- provides OSG a way to contact site personnel (tickets, security notices)
- adds the site and the site administrators in the "OSG database"
- adds the resources in the information system, BDII, and in the MyOSG monitoring.
Follow the RegistrationInstructions
to register your resource.
- 20 May 2010