Compute Element Post Install

Choosing and Installing a CA Distribution

OSG 1.0.0 no longer automatically pulls CAs.

To pull the OSG recommended CA distribution edit the cacerts_url in the configuration file at


This file contains URLs to CA Certificate distributions including the OSG GOC distribution with certificates recommended by the OSG Security Team, as well as the VDT convenience distribution. You must uncomment one of these (or create your own), and then run the commands below to activate the certificate updates.

# source $VDT_LOCATION/; $VDT_LOCATION/vdt/sbin/vdt-setup-ca-certificates

# vdt-control --enable vdt-update-certs
# vdt-control --on vdt-update-certs

To read a summary of the security team's CA distribution process see

To view details of the content of the OSG recommended certificate package see

Included topic: CESoft Ware

Discovering exactly what software has been installed

# source $VDT_LOCATION/ 
# ./osg-version 
ITB 0.9.2

# source $VDT_LOCATION/ 
# vdt-version 
You have installed a subset of VDT version 1.8.1:
    Apache HTTPD 2.2.4
    gLite CEMon Server (INFN release from 2006-05-19, plus RAW dialect) 1.7.1
    CA Certificates v30 (includes IGTF 1.16 CAs)
    Condor/Condor-G Developers Series 6.9.4
    EDG Make Gridmap 2.9.0
    Fetch CRL 2.6.2
    Generic Information Provider 1.0.15 (Iowa 15-Feb-2006)
    Globus Toolkit, pre web-services, client 4.0.5
    Globus Toolkit, pre web-services, server 4.0.5
    Globus Toolkit, web-services, client 4.0.5
    Globus Toolkit, web-services, server 4.0.5
    GLUE Schema 1.2 draft 7
    GPT 3.2
    Gratia Condor Probe 0.27.3-1
    Gratia Metric Probe 0.27.3-1
    Java SDK 1.4.2_14
    Java 5 SDK 1.5.0_12
    KX509 20031111
    Logrotate 3.7
    MonALISA 1.6.22
    MyProxy 3.9
    MySQL 4.1.22
    MySQL Connector/J 5.0.6
    OSG Resource and Service Validation 1.4
    Pegaus Worker Package 2.0.1
    PPDG Cert Scripts 2.5
    PRIMA Authorization Module 0.7
    PRIMA Authorization Module For GT4 Web Services 0.1.2
    RLS, client 3.0.041021
    SRM V1 Client 1.25
    SRM V2 Client
    syslog-ng 2.0.4
    Apache Tomcat 5.0.28
    UberFTP 1.24
    Wget 1.10.2

Finding all services that are part of your install

You can use the vdt-control command to see all services that are installed as part of the install, and whether or not they are enabled or disabled. For example:

# vdt-control --list 
Service            | Type   | Desired State
fetch-crl          | cron   | enable
vdt-rotate-logs    | cron   | enable
gris               | init   | do not enable
globus-gatekeeper  | inetd  | enable
gsiftp             | inetd  | enable
mysql              | init   | enable
globus-ws          | init   | enable
edg-mkgridmap      | cron   | do not enable
gums-host-cron     | cron   | do not enable   
MLD                | init   | do not enable           
vdt-update-certs   | cron   | do not enable    
condor-devel       | init   | enable                  
osg-rsv            | init   | enable                     
apache             | init   | enable
tomcat-5           | init   | enable
syslog-ng          | init   | enable                     
gratia-condor      | cron   | enable

For debugging purposes only, you can look at $VDT_LOCATION/vdt/services/state. This file is automatically generated and should not be edited by hand. It contains all the information used by vdt-control to start and stop services in the VDT.

Below we break down the services that you can expect to be on your system. All of these are listed by vdt-control: they are broken down here in order to give you more information. Note that the corresponding files will not be moved in /etc/xinetd.d, crontab, or /etc/init.d until the vdt-control --on command is executed, and then only for those that are marked "enable" in the list of services above.

The xinetd services that are included

Two services are run via xinetd:
Service Description
gsiftp GridFTP
globus-gatekeeper The GRAM 2 (pre-web services) interface


Two entries have been appended to your /etc/services file:
Service Port
gsiftp 2811/tcp
globus-gatekeeper 2119/tcp

The init.d services that are included in /etc/init.d

Service Description
globus-ws Globus web services
mysql used by globus-ws
condor setup optionally by ManagedFork or Globus-Condor-Setup
MLD MonALISA - setup optionally by
tomcat-5 used by CEMon and BDII and optionally globus-ws
apache In front of Tomcat
condor-devel A new version of condor that is used only to manage the crons from the RSV process
osg-rsv Resource and Service Validation probes
syslog-ng optional service to share syslog-ng logs with central ITB staff for debugging purposes

Cron jobs

Some cron jobs are installed with random times. Look at crontab -l to see which ones you have. Likely candidates are:

Cron Job Description
fetch-crl Update the certificate authority revocation lists. Verify importation for security purposes
vdt-rotate-logs Rotate log files periodically, to keep them from growing too large
edg-mkgridmap If you use a grid-mapfile, keeps it up to date
gums-host-cron If you use GUMS, generate the osg-user-vo-map.txt file
gratia-condor Report accounting data from Condor
gratia-pbs-lsf Report accounting data from PBS or LSF
vdt-update-certs optional way to get new CA-Certificates packages without doing pacman -update

Log file rotation

The following is an example of the log files that are configured for rotation. To see exactly what files are rotated in your installation, look in $VDT_LOCATION/vdt/etc/vdt.logrotate:

# Rotate daily
# Keep a week's worth
rotate 7
# Compress old files
compressoptions -9f
# Truncate, instead of moving, files to avoid problems with apps
# Just ignore it if they're not there
# Rotate these files
/storage/local/data1/osg-ce/globus/var/grid-info-system.log {}
/storage/local/data1/osg-ce/globus/var/accounting.log {}
/storage/local/data1/osg-ce/globus/var/globus-gatekeeper.log {}
/storage/local/data1/osg-ce/globus/var/log/gridftp.log {}
/storage/local/data1/osg-ce/globus/var/log/gridftp-auth.log {}
/storage/local/data1/osg-ce/apache/logs/access_log {}
/storage/local/data1/osg-ce/apache/logs/error_log {}
/storage/local/data1/osg-ce/apache/logs/mod_jk.log {}
/storage/local/data1/osg-ce/apache/logs/ssl_request_log {}
/storage/local/data1/osg-ce/tomcat/v5/logs/catalina.out {}

Included topic: Compute Element Firewalls


Grid components are distributed throughout the network, and services such as gatekeepers and data movement utilities are required to be accessible to the dynamic cloud of clients and peer services. This distributed and dynamic requirement places the burden of the security on the implementation of the application.

Due to the discovery of significant vulnerabilities in recent years, network-based applications are untrusted by default. To solve the application problem, effort has focused on developing and deploying firewalls, which restricts full and free network access. (You might say that this is analogous to building a house with no doors. Is it safe? Yes. Is it useful? No.)

Some essential network-based applications have been "hardened," such as mail relay services, web servers, and secure shell daemons. These are further protected further by IP address filtering to prevent access from unknown hosts or domains.

Grid components which are located behind a network firewall face special challenges for Grid setup and operations.

There are two styles of firewalls usually encountered.

  • A network firewall which is upstream from your server (usually centrally maintained). This blocks all traffic to your host.
  • Host-based firewalls which are set up and maintained by individual host administrators. These are usually set up and configured by the iptables program which filters incoming network packets which arrive for the host.

In addition to host-based firewalls, hosts can choose to implement host based access rules (usually set up with the tcp_wrapper or hosts_allow utilities).

Network traffic can be blocked at the firewall for both incoming and outgoing dataflow depending on hostnames, ip addresses, ports and protocols.

A common setup is to allow any outgoing connection, while significantly (if not completely) restricting incoming connections. The Globus project provides thorough documentation, including but not limited to the Globus Toolkit firewall requirements document.

You may need to open ports on your firewall for your batch system (Condor, LSF, etc.), too! See, for example, information about Condor's use of ports.

Port ranges

Port usage which may require firewall updates:

  • Inbound:
    • GRAM: 2119/tcp inbound
    • GridFTP: 2811/tcp inbound
    • MDS: 2135/tcp inbound (obsolete)
    • Web Services: 9443/tcp inbound
    • GRAM callback: user-defined contiguous range (set via the environment variable GLOBUS_TCP_PORT_RANGE=beginport,endport). It should span at least 100 ports for a small site.
    • Monalisa: 9000/udp (for ABping measurements). These are specified in the file $VDT_LOCATION/MonaLisa/Service/VDTFarm/
  • Outbound:
    • GRAM callback: user-defined contiguous range (set via the environment variable GLOBUS_TCP_SOURCE_RANGE=beginport,endport).

GRAM and GridFTP need to know the port range that you've opened up via environment variables. You need to set the GLOBUS_TCP_PORT_RANGE=beginport,endport for inbound ports. If you restrict outbound connections, you will also need to set GLOBUS_TCP_SOURCE_RANGE=beginport,endport. These may be set in $VDT_LOCATION/vdt/etc/vdt-local-setup.[c]sh. As of VDT 1.6.x, you should NOT set the allowed port ranges by editing the xinetd configuration files — the examples below do imply such editing, but will not survive a vdt-control off and on cycle.

The variables applied in the local setup as above will be used by GRAM, GridFTP, and any clients that require them.

The above ports and protocols must be open to and from all grid clients and server machines participating in the grid in order to provide minimal functionality.

In addition to the above, port 9443 must be open for both incoming and outgoing connections in order to test the web-services capabilities of the most recent versions of the VDT.

You also may need to open the following optional incoming ports for additional Grid services. Note that unlike the ones listed above, the following are optional and are only needed if you are running those specific services or if required by your specific virtual organization.

  • GIIS: 2136/tcp (obsolete)
  • GSISSH: 22/tcp
  • MyProxy: 7512/tcp
  • VOMS-Admin and/or GUMS: 8443/tcp
  • VOMS 15001/tcp (ports start at 15001 and increment by 1 for each VO you support, used for voms-proxy-init service).
  • RLS server: 39281/tcp
  • Squid server: 3128/tcp

Here is a sample of the a /etc/hosts.allow with the GLOBUS services opened:

  # hosts.allow   This file describes the names of the hosts which are
  #               allowed to use the local INET services, as decided
  #               by the '/usr/sbin/tcpd' server.
  ALL : localhost : ALL : ALL

iptables for RHEL or compatible systems

The default firewall configuration for Red Hat's iptables sets the system up with a stateful packet filter. This is different than some legacy RH7 systems as by default no ports that are not explicitly opened by the iptables script will be open. This includes high numbered ports that are often used by grid services.

If your preference is to leave as much of the stateful packet filtering in place but enable just those grid services you want to deploy, then you can use the following instructions.

Two changes need to be made to an OSG gateway with a host based iptables stateful firewall.

First is the configuration of the firewall itself. On RHEL or similar systems this is done in /etc/sysconfig/iptables.

The Chain RH-Firewall-1-INPUT is a default chain for RHEL3. This chain is also sometimes called INPUT. Make sure the following rules use the chain that other rules in /etc/sysconfig/iptables do.

For GSISSH this port is often already open for systems. You can use either this rule or the default rule setup at install time if you selected custom firewall and enabled ssh.

# Globus: Requires addition configuration in /etc/xinetd.d/globus-gatekeeper
# set: env = GLOBUS_TCP_PORT_RANGE=40000,50000
# This allows up to 10,000 ports and matches the globus config.
# How globus is configured to use these ports is subject to change in an upcoming
# release
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 40000:50000 -j ACCEPT
# Monalisa, grabs 3 ports from the following range
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 9000:9010 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p ucp -m ucp --dport 9000 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2119 -j ACCEPT
# Gridftp
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2811 -j ACCEPT

# Optional Services
# RLS Server
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 39281 -j ACCEPT
# MyProxy
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 7512 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2135 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 2136 -j ACCEPT
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 8443 -j ACCEPT
# WebServices
-A RH-Firewall-1-INPUT  -m state --state NEW -p tcp -m tcp --dport 9443 -j ACCEPT

Second, we configure Globus to use the allowed inbound port range. Do NOT do this by hand-editing the xinetd.d files in versions equal to or later than VDT 1.6.x (ITB 0.5.x), as these will not survive a vdt-control off and on cycle (the xinetd.d files are rewritten by such a cycle). Instead, set them in the vdt-local-setup.[c]sh files. $VDT_LOCATION/vdt/etc/

setenv GLOBUS_TCP_SOURCE_RANGE 40000,49150
setenv GLOBUS_TCP_PORT_RANGE 49000,49150

If you limit the globus-related port range to certain values, it may be necessary to adjust the Linux ephemeral port range to avoid these values. If this has not already been done, check the /etc/sysctl.conf for the following lines and insert if needed:

# Limit ephemeral ports to avoid globus tcp port range
# See OSG CE install guide
net.ipv4.ip_local_port_range = 10240 39999

Save and exit if edited. Then, if you changed it, apply the changes by doing: sysctl -p

After editing the above files run the following commands

# /etc/rc.d/init.d/iptables restart 
  Flushing firewall rules:                                   [  OK  ]
  Setting chains to policy ACCEPT: filter                    [  OK  ]
  Unloading iptables modules:                                [  OK  ]
  Applying iptables firewall rules:                          [  OK  ]
# /etc/rc.d/init.d/xinetd reload 
  Reloading configuration:                                   [  OK  ] 

More information about Globus firewall requirements

Please look on the Globus website for more information on firewalls.

Included topic: Register Resource With GOC

To register the resource with the OSG Grid Operations Center and into the Grid Catalog please use the GOC's OIM Registration DB. To use OIM you will need to have an IGTF member certificate in your browser. If you have previously registered your resource with the OSG please visit this link and confirm or update any stale information about your resource. If you are registering with the OSG for the first time you will need to fill in User Contact information before registering your resource. The OIM pages will guide you through the process.

A minimal amount of information is needed for the OSG Grid Operations Center (GOC) to publish a resource to the monitoring and operational infrastructure. Please take time to fill out the registration forms as completely as possible, this will allow the GOC to keep track of the resource and contact information and gather updated information from you as necessary.

Complete: 3
Responsible: Main.StevenTimm - 18 Oct 2007
Reviewer - date:

Topic revision: r13 - 09 Jan 2012 - 00:39:22 - AnandPadmanabhan

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..