Compute Element Post Install
Choosing and Installing a CA Distribution
OSG 1.0.0 no longer automatically pulls CAs.
To pull the OSG recommended CA distribution edit the cacerts_url in the configuration file at
This file contains URLs to CA Certificate distributions including the OSG GOC distribution with certificates recommended by the OSG Security Team, as well as the VDT convenience distribution. You must uncomment one of these (or create your own), and then run the commands below to activate the certificate updates.
# source $VDT_LOCATION/vdt-questions.sh; $VDT_LOCATION/vdt/sbin/vdt-setup-ca-certificates
# vdt-control --enable vdt-update-certs
# vdt-control --on vdt-update-certs
To read a summary of the security team's CA distribution process see https://twiki.grid.iu.edu/bin/view/Documentation/CaDistribution
To view details of the content of the OSG recommended certificate package see http://software.grid.iu.edu/pacman/cadist/
Included topic: CESoft Ware
# source $VDT_LOCATION/setup.sh
# source $VDT_LOCATION/setup.sh
You have installed a subset of VDT version 1.8.1:
Apache HTTPD 2.2.4
gLite CEMon Server (INFN release from 2006-05-19, plus RAW dialect) 1.7.1
CA Certificates v30 (includes IGTF 1.16 CAs)
Condor/Condor-G Developers Series 6.9.4
EDG Make Gridmap 2.9.0
Fetch CRL 2.6.2
Generic Information Provider 1.0.15 (Iowa 15-Feb-2006)
Globus Toolkit, pre web-services, client 4.0.5
Globus Toolkit, pre web-services, server 4.0.5
Globus Toolkit, web-services, client 4.0.5
Globus Toolkit, web-services, server 4.0.5
GLUE Schema 1.2 draft 7
Gratia Condor Probe 0.27.3-1
Gratia Metric Probe 0.27.3-1
Java SDK 1.4.2_14
Java 5 SDK 1.5.0_12
MySQL Connector/J 5.0.6
OSG Resource and Service Validation 1.4
Pegaus Worker Package 2.0.1
PPDG Cert Scripts 2.5
PRIMA Authorization Module 0.7
PRIMA Authorization Module For GT4 Web Services 0.1.2
RLS, client 3.0.041021
SRM V1 Client 1.25
SRM V2 Client 22.214.171.124
Apache Tomcat 5.0.28
You can use the
command to see all services that are installed as part of the install, and whether or not they are enabled or disabled. For example:
# vdt-control --list
Service | Type | Desired State
fetch-crl | cron | enable
vdt-rotate-logs | cron | enable
gris | init | do not enable
globus-gatekeeper | inetd | enable
gsiftp | inetd | enable
mysql | init | enable
globus-ws | init | enable
edg-mkgridmap | cron | do not enable
gums-host-cron | cron | do not enable
MLD | init | do not enable
vdt-update-certs | cron | do not enable
condor-devel | init | enable
osg-rsv | init | enable
apache | init | enable
tomcat-5 | init | enable
syslog-ng | init | enable
gratia-condor | cron | enable
For debugging purposes only
, you can look at
. This file is automatically generated and should not
be edited by hand. It contains all the information used by vdt-control to start and stop services in the VDT.
Below we break down the services that you can expect to be on your system. All of these are listed by
: they are broken down here in order to give you more information. Note that the corresponding files will not be moved in /etc/xinetd.d, crontab, or /etc/init.d until the vdt-control --on command is executed, and then only for those
that are marked "enable" in the list of services above.
The xinetd services that are included
Two services are run via xinetd:
|| The GRAM 2 (pre-web services) interface
Two entries have been appended to your
The init.d services that are included in /etc/init.d
|| Globus web services
|| used by globus-ws
|| setup optionally by ManagedFork or Globus-Condor-Setup
|| MonALISA - setup optionally by configure-osg.py
|| used by CEMon and BDII and optionally globus-ws
|| In front of Tomcat
|| A new version of condor that is used only to manage the crons from the RSV process
|| Resource and Service Validation probes
|| optional service to share syslog-ng logs with central ITB staff for debugging purposes
Some cron jobs are installed with random times. Look at
to see which ones you have. Likely candidates are:
| Cron Job
|| Update the certificate authority revocation lists. Verify importation for security purposes
|| Rotate log files periodically, to keep them from growing too large
|| If you use a grid-mapfile, keeps it up to date
|| If you use GUMS, generate the osg-user-vo-map.txt file
|| Report accounting data from Condor
|| Report accounting data from PBS or LSF
|| optional way to get new CA-Certificates packages without doing pacman -update
The following is an example of the log files that are configured for rotation. To see exactly what files are rotated in your installation, look in
# Rotate daily
# Keep a week's worth
# Compress old files
# Truncate, instead of moving, files to avoid problems with apps
# Just ignore it if they're not there
# Rotate these files
Included topic: Compute Element Firewalls
Grid components are distributed throughout the network, and services such as gatekeepers and data movement utilities are required to be accessible to the dynamic cloud of clients and peer services. This distributed and dynamic requirement places the burden of the security on the implementation of the application.
Due to the discovery of significant vulnerabilities in recent years, network-based applications are untrusted by default. To solve the application problem, effort has focused on developing and deploying firewalls, which restricts full and free network access. (You might say that this is analogous to building a house with no doors. Is it safe? Yes. Is it useful? No.)
Some essential network-based applications have been "hardened," such as mail relay services, web servers, and secure shell daemons. These are further protected further by IP address filtering to prevent access from unknown hosts or domains.
Grid components which are located behind a network firewall face special challenges for Grid setup and operations.
There are two styles of firewalls usually encountered.
- A network firewall which is upstream from your server (usually centrally maintained). This blocks all traffic to your host.
- Host-based firewalls which are set up and maintained by individual host administrators. These are usually set up and configured by the
iptables program which filters incoming network packets which arrive for the host.
In addition to host-based firewalls, hosts can choose to implement host based access rules (usually set up with the
Network traffic can be blocked at the firewall for both incoming and outgoing dataflow depending on hostnames, ip addresses, ports and protocols.
A common setup is to allow any outgoing connection, while significantly (if not completely) restricting incoming connections. The Globus project provides thorough documentation, including but not limited to the Globus Toolkit firewall requirements
- You may need to open ports on your firewall for your batch system (Condor, LSF, etc.), too! See, for example, information about Condor's use of ports.
Port usage which may require firewall updates:
- GRAM: 2119/tcp inbound
- GridFTP: 2811/tcp inbound
- MDS: 2135/tcp inbound (obsolete)
- Web Services: 9443/tcp inbound
- GRAM callback: user-defined contiguous range (set via the environment variable
GLOBUS_TCP_PORT_RANGE=beginport,endport). It should span at least 100 ports for a small site.
- Monalisa: 9000/udp (for ABping measurements). These are specified in the file
- GRAM callback: user-defined contiguous range (set via the environment variable
GRAM and GridFTP
need to know the port range that you've opened up via environment variables. You need to set the
for inbound ports. If you restrict outbound connections, you will also need to set
. These may be set in
. As of VDT 1.6.x, you should NOT set the allowed port ranges by editing the xinetd configuration files — the examples below do imply such editing, but will not survive a vdt-control off and on cycle.
The variables applied in the local setup as above will be used by GRAM, GridFTP
, and any clients that require them.
The above ports and protocols must be open
to and from all grid clients and server machines participating in the grid in order to provide minimal functionality.
In addition to the above, port 9443 must be open for both incoming and outgoing connections in order to test the web-services capabilities of the most recent versions of the VDT.
You also may need to open the following optional incoming ports for additional Grid services. Note that unlike the ones listed above, the following are optional
and are only needed if you are running those specific services or if required by your specific virtual organization.
- GIIS: 2136/tcp (obsolete)
- GSISSH: 22/tcp
- MyProxy: 7512/tcp
- VOMS-Admin and/or GUMS: 8443/tcp
- VOMS 15001/tcp (ports start at 15001 and increment by 1 for each VO you support, used for voms-proxy-init service).
- RLS server: 39281/tcp
- Squid server: 3128/tcp
Here is a sample of the a
with the GLOBUS services opened:
# hosts.allow This file describes the names of the hosts which are
# allowed to use the local INET services, as decided
# by the '/usr/sbin/tcpd' server.
ALL : localhost
vdt-run-gsiftp.sh : ALL
vdt-run-globus-gatekeeper.sh : ALL
The default firewall configuration for Red Hat's iptables sets the system up with a stateful packet filter. This is different than some legacy RH7 systems as by default no ports that are not explicitly opened by the iptables script will be open. This includes high numbered ports that are often used by grid services.
If your preference is to leave as much of the stateful packet filtering in place but enable just those grid services you want to deploy, then you can use the following instructions.
Two changes need to be made to an OSG gateway with a host based iptables stateful firewall.
First is the configuration of the firewall itself. On RHEL or similar systems this is done in
The Chain RH-Firewall-1-INPUT is a default chain for RHEL3. This chain is also sometimes called INPUT. Make sure the following rules use the chain that other rules in
- For GSISSH this port is often already open for systems. You can use either this rule or the default rule setup at install time if you selected custom firewall and enabled ssh.
# Globus: Requires addition configuration in /etc/xinetd.d/globus-gatekeeper
# set: env = GLOBUS_TCP_PORT_RANGE=40000,50000
# This allows up to 10,000 ports and matches the globus config.
# How globus is configured to use these ports is subject to change in an upcoming
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 40000:50000 -j ACCEPT
# Monalisa, grabs 3 ports from the following range
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 9000:9010 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p ucp -m ucp --dport 9000 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 2119 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 2811 -j ACCEPT
# Optional Services
# RLS Server
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 39281 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 7512 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 2135 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 2136 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 8443 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -p tcp -m tcp --dport 9443 -j ACCEPT
Second, we configure Globus to use the allowed inbound port range. Do NOT do this by hand-editing the xinetd.d files in versions equal to or later than VDT 1.6.x (ITB 0.5.x), as these will not survive a vdt-control off and on cycle (the xinetd.d files are rewritten by such a cycle). Instead, set them in the
setenv GLOBUS_TCP_SOURCE_RANGE 40000,49150
setenv GLOBUS_TCP_PORT_RANGE 49000,49150
If you limit the globus-related port range to certain values, it may be necessary to adjust the Linux ephemeral port range to avoid these values. If this has not already been done, check the
for the following lines and insert if needed:
# Limit ephemeral ports to avoid globus tcp port range
# See OSG CE install guide
net.ipv4.ip_local_port_range = 10240 39999
Save and exit if edited. Then, if you changed it, apply the changes by doing:
After editing the above files run the following commands
# /etc/rc.d/init.d/iptables restart
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
Applying iptables firewall rules: [ OK ]
# /etc/rc.d/init.d/xinetd reload
Reloading configuration: [ OK ]
Please look on the Globus
website for more information on firewalls.
Included topic: Register Resource With GOC
To register the resource with the OSG Grid Operations Center and into the Grid Catalog please use the GOC's OIM Registration DB
. To use OIM you will need to have an IGTF
member certificate in your browser. If you have previously registered your resource with the OSG please visit this link and confirm or update any stale information about your resource. If you are registering with the OSG for the first time you will need to fill in User Contact information before registering your resource. The OIM pages will guide you through the process.
A minimal amount of information is needed for the OSG Grid Operations Center (GOC) to publish a resource to the monitoring and operational infrastructure. Please take time to fill out the registration forms as completely as possible, this will allow the GOC to keep track of the resource and contact information and gather updated information from you as necessary.
Responsible: Main.StevenTimm - 18 Oct 2007
Reviewer - date: