OSG Services and Firewalls

About this Document

hand This document is for System Administrators. It starts with an introduction on Firewalls and how those terms are translated for OSG software. Then a brief description of services provided by a Compute Element and Storage Element will be given, followed by recommendations how to adjust the firewall for their correct operation. Servers have to worry about several ports while OSG Client installations only have to worry about the GLOBUS_TCP_PORT_RANGE, GLOBUS_TCP_SOURCE_RANGE and the HTCondor ports (LOWPORT, HIGHPORT).

This document also defines a Firewall table that can be included in other documents to specify the network requirements. You can find more about it in the formatting document.

To use the table, list the lines you desire in the table in the variable "lines", separated by commas with no spaces.

All the predefined lines are: gram,portrange,portsource,gridftp,srm,srm2,gsissh,voms,vomsadmin,gums,myproxy,squid,squidmonitor,condorcollector,condor,rsvin,rsvout,various

You can add custom lines using a backslash "\" after the include and adding your lines (remember the 2 "space" on each side to center the cell). This prints the full table and a custom line:

%INCLUDE{"Documentation/Release3.FirewallInformation" section="FirewallTable"  lines="gram,portrange,portsource,gridftp,srm,srm2,gsissh,voms,gums,myproxy,squid,squidmonitor,condorcollector,condor,rsvin,rsvout,various"}% \
| HTTP |  tcp  |  80  |  %ICON{choice-yes}%  | | My custom HTTP server with special explanation |
That results in:

Service Name Protocol Port Number Inbound Outbound Comment
GRAM tcp 2119 Y    
GRAM callback tcp GLOBUS_TCP_PORT_RANGE Y   contiguous range of ports
GRAM callback tcp GLOBUS_TCP_SOURCE_RANGE   Y contiguous range of ports
GridFTP tcp 2811 and GLOBUS_TCP_SOURCE_RANGE Y   contiguous range of ports
Storage Resource Manager tcp 8080 Y    
Storage Resource Manager tcp 8443 Y    
GSISSH tcp/udp 22 or 2222 Y    
MyProxy tcp 7512 Y    
GUMS tcp 8443 Y    
VOMS tcp 15001+ Y   range of ports, increment by 1 for each VO supported
Squid tcp 3128 Y Y Also limited in squid ACLs. Both in and outbound must not be wide open to internet simultaneously
Squid monitor udp 3401 Y   Also limited in squid ACLs. Should be limited to monitoring server addresses
HTCondor collector tcp 9618 Y   HTCondor Collector (received ClassAds from resources and jobs)
HTCondor port range tcp LOWPORT, HIGHPORT Y   contiguous range of ports
HTTP tcp 80 Y   RSV runs an HTTP server (Apache) that publishes a page with the RSV testing results
HTTP tcp 80   Y RSV pushes testing results to the OSG Gratia Collectors at opensciencegrid.org
various various various   Y Allow outbound network connection to all services that you want to test
HTTP tcp 80 choice-yes   My custom HTTP server with special explanation

Introduction

Network traffic may be blocked by a firewall for inbound and outbound traffic in dependence on host- and domain names, IP addresses, port numbers and protocols. In this document we distinguish between two types of firewalls:

  1. network firewalls administrated centrally and typically outside of your administrative domain
  2. host-based firewalls within your administrative domain

The protocol and port requirements for OSG software components are listed in the install documents and summarized below. Please provide them to the administrator of a network firewall potentially blocking network traffic to your host. Next, follow the instructions to adjust the firewall settings on your host-based firewall.

Introduction to firewalls

If you are familiar with ports and Firewalls you can skip this section. An IP connection is identified by an address and ports. Here are some clarifications on the way ports are used on a host.

Let's assume to be on a host. Servers require incoming connections to one or more well known port (port known a priori by the client contacting the server on that host) Then both clients and servers may require a variable number of incoming or outgoing connections on arbitrary (ephemeral) ports. If there is any kind of firewall allowing only limited connectivity, then the processes must have a known set of outbound ephemeral ports (ephemeral ports that are allowed outbound connectivity) they can choose from to connect outside, and must have a known set of inbound ephemeral ports (ephemeral ports that are allowed inbound connectivity) they can choose from to listen for callbacks.

If a process on a host (client or server) is behind a firewall that is not transparent but is a NAT router or a proxy, e.g when the client/server resides on a private network like in Figure 2, then the set of inbound ephemeral ports is the one opened on the firewall (router/proxy), the address for the callbacks has to be the public address (probably the one of the router/proxy) and the router has to be configured to perform port forwarding both of the well known ports of servers behind it and of all the inbound ephemeral ports. As a consequence of this setup, all clients and servers on different hosts behind the same NAT router must use disjunct sets of ports (well known ports + inbound ephemeral ports).

Figure 1 shows a client (b) and a server (a) behind a host firewall or a transparent network firewall.

Figure 2 shows a client and a server on a private network behind a NAT router. Red borders indicate port forwarding. Note the position of the transversal mark (dark blue/green) on the ephemeral ports indicating where the port range is defined: outbound ports are on the host, inbound ports are on the NAT router (and can be different form the ports on the host they are forwarded to) because are the ones used by the program calling back the client/server, coming form the public network.

Figure 1a:
firewall-server-12p.png

Figure 1b:
firewall-client-12p.png

          Figure 2:
firewall-net1-12p.png

Configuring the OSG software

The software installed/configured by OSG RPMs includes Globus, HTCondor, Web servers (Apache, Tomcat). These need to know Firewall related information and here is a summary of how to specify it.

A Firewall:

  • must open (or forward) the well known ports: the port number is mentioned in the network requirements for that specific software (see its install document).
  • can be configured with a range of inbound ephemeral ports and a range of outbound ephemeral ports that can be used by the software, depending on the software and network configuration.
The software needs to be made aware of the network specifics (ports and host name) using its own mechanisms: generally configuration values or environment variables defined at the startup. Here is a summary:

Ports Globus HTCondor Tomcat Apache
well known ports varies values in condor_config (1) proxyPort (2) Listen in httpd.conf (3)
inbound ephemeral ports GLOBUS_TCP_PORT_RANGE HIGHPORT, LOWPORT    
outbound ephemeral ports GLOBUS_TCP_SOURCE_RANGE HIGHPORT, LOWPORT    
public host name GLOBUS_NAME values in condor_config (1) proxyName (2) (3)

Notes:

  1. HTCondor configuration files depend on ist installation and configuration, e.g. /etc/condor/condor_config. Generally host name and port are specified in the same entry, e.g. COLLECTOR_HOST = $(CONDOR_HOST): 9618, and CONDOR_HOST = public_name.domain . See HTCondor's manual
  2. Tomcat's configuration file is /etc/tomcat5/server.xml (/etc/tomcat6/server.xml on EL6). See Tomcat's documentation
  3. Apache's main configuration file is /etc/httpd/conf/httpd.conf. The configuration may involve virtual hosts, proxy and reverse proxy functionalities. See Apache's documentation

HELP NOTE
Host (and service) certificates must refer always to the public name, the one of the NAT router or the proxy if the host is behind them.

Requirements

  1. be familiar with your institute's network policy and firewall configuration
  2. root access is required to configure a host-based firewall using iptables

OSG software network requirements

GRAM Compute Element Services and Firewall Requirements

Service Name Protocol Port Number Inbound Outbound Comment
GRAM tcp 2119 Y    
GRAM callback tcp GLOBUS_TCP_PORT_RANGE Y   contiguous range of ports
GRAM callback tcp GLOBUS_TCP_SOURCE_RANGE   Y contiguous range of ports
GridFTP tcp 2811 and GLOBUS_TCP_SOURCE_RANGE Y   contiguous range of ports

For GLOBUS_TCP_PORT_RANGE is recommended to open 8 ports*number of job slots. Please note: This number is wrong and we will update is soon (July 2012).
Allow inbound and outbound network connection to all cluster servers, e.g. GUMS and job manager head-node
Inbound and outbound network connection outside of the cluster can be limited to any clients who may need to submit jobs

HTCondor-CE Services and Firewall Requirements

Service Name Protocol Port Number Inbound Outbound Comment
HTCondor-CE port tcp 9619 Y   Used to locate HTCondor-CE daemons
HTCondor-CE shared_port daemon port tcp 9620 Y   Only for HTCondor-CE < 8.3.2: Used for aggregating ephemeral ports used by HTCondor into a single network port

Allow inbound and outbound network connection to all internal site servers, such as GUMS and the batch system head-node only ephemeral outgoing ports are necessary.

Storage Element Services and Firewall Requirements

Service Name Protocol Port Number Inbound Outbound Comment
GridFTP tcp 2811 and GLOBUS_TCP_SOURCE_RANGE choice-yes   contiguous range of ports
GRAM callback tcp GLOBUS_TCP_PORT_RANGE choice-yes   contiguous range of ports
GRAM callback tcp GLOBUS_TCP_SOURCE_RANGE   choice-yes contiguous range of ports
Storage Resource Manager tcp 8080 choice-yes    
Storage Resource Manager tcp 8443 choice-yes    

Client Services and Firewall Requirements

Service Name Protocol Port Number Inbound Outbound Comment
GRAM callback tcp GLOBUS_TCP_PORT_RANGE Y   contiguous range of ports
GRAM callback tcp GLOBUS_TCP_SOURCE_RANGE   Y contiguous range of ports
HTCondor port range tcp LOWPORT, HIGHPORT Y   contiguous range of ports

GRAM is not really a service on the client. It is the protocol used by the Globus clients. Anyway the clients still requires the port ranges to be open: job submission needs ports to reach the servers and to transfer back the output; file transfers need ports for control and data sessions.
HTCondor is in reality HTCondor-G the version configured to submit grid jobs.

Other Optional Services and Firewall Requirements

You may need to open the following required inbound ports for listed optional services.

Service Name Protocol Port Number Inbound Outbound Comment
GSISSH tcp/udp 22 or 2222 Y    
MyProxy tcp 7512 Y    
GUMS tcp 8443 Y    
VOMS tcp 15001+ Y   range of ports, increment by 1 for each VO supported
Squid tcp 3128 Y Y Also limited in squid ACLs. Both in and outbound must not be wide open to internet simultaneously
Squid monitor udp 3401 Y   Also limited in squid ACLs. Should be limited to monitoring server addresses

BOSCO Firewall

on

Gratia Service (aka Collector) Firewall

Gratia Service

These firewall requirements are for the host running the Gratia Service using Apache Tomcat.

Service Name Protocol Port Number Inbound Outbound Comment
Gratia Service tcp 8443 Y   Gratia Service (which runs within Tomcat) web interface. It must be available to all the probes reporting data, all servers receiving or sending replication data and the database

MySQL Database

The firewall requirement for the database:

Service Name Protocol Port Number Inbound Outbound Comment
MySQL tcp/udp 3306 Y   3306 is the default MySQL port. Can be configured to run on a different port. Can also communicate via a named pipe if the client is on the same host as the server

HELP NOTE
The port of the database can change as long as the database and the Gratia Service use the same port and it may also be avoided if service and database are on the same host and use a named pipe to communicate.

Host-based Firewall Configuration

The default firewall configuration for RedHat's iptables defines a stateful packet filter; no ports that are not explicitly opened by iptables will be open. This includes high numbered ports that are often used by grid services.

If your preference is to leave as much of the stateful packet filtering in place but enable just those grid services you want to deploy, then you can use the following instructions. On RHEL or similar systems the firewall is configured using the /etc/sysconfig/iptables file:

(You might want to change the name to be something more appropriate on your system, such as RH-Firewall-1-INPUT on Red Hat Enterprise Linux systems or INPUT on Scientific Linux systems. Model the rules after the ones already existing in your iptables file.)

# GLOBUS_TCP_PORT_RANGE
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport <begin port>:<end port> -j ACCEPT
# GRAM
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport 2119 -j ACCEPT
# Gridftp
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport 2811 -j ACCEPT
# Optional Services
# MyProxy
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport 7512 -j ACCEPT
# GSISSH/SSH
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport 22 -j ACCEPT
-A OSG-INPUT  -m state --state NEW -p udp -m tcp --dport 22 -j ACCEPT
# GUMS/VOMS
-A OSG-INPUT  -m state --state NEW -p tcp -m tcp --dport 8443 -j ACCEPT

HELP NOTE
Please open the range of Globus Ports? according to your Compute Element configuration.

Finally restart the firewall as the root user for changes to take effect:

[root@ce ~]$ /etc/rc.d/init.d/iptables restart
  Flushing firewall rules:                                   [  OK  ]
  Setting chains to policy ACCEPT: filter                    [  OK  ]
  Unloading iptables modules:                                [  OK  ]
  Applying iptables firewall rules:                          [  OK  ]

[root@ce ~]$ /etc/rc.d/init.d/xinetd reload
  Reloading configuration:                                   [  OK  ] 

Stateful Firewalls

A Stateful Firewall keeps track of the state of network connections. If a TCP connection is closed unexpectedly a stateful firewall will not know that it has been closed. In this case the operating system may allow programs to reuse the port number, but the firewall will not.

To avoid using the same ports too quickly, and hopefully avoid the situation where a stateful firewall will not allow a port to be reused. The Globus Toolkit keeps a record of TCP ports using two files which are defined by two environment variables.

You should create /etc/profile.d/globus_firewall.sh, similar to what's below. (A future version of osg-configure will do this for you.)

# Set GLOBUS_TCP_PORT_RANGE_STATE_FILE to the location where Globus should record
# TCP port usage for outbound connections in case of a stateful firewall.
export GLOBUS_TCP_PORT_RANGE_STATE_FILE=<location on file system>

# Set GLOBUS_TCP_SOURCE_RANGE_STATE_FILE to the location where Globus should record
# TCP port usage for inbound connections in case of a stateful firewall.
export GLOBUS_TCP_SOURCE_RANGE_STATE_FILE=<location on file system>

and for the TCShell in /etc/profile.d/globus_firewall.csh :

# Set GLOBUS_TCP_PORT_RANGE_STATE_FILE to the location where Globus should record
# TCP port usage for outbound connections in case of a stateful firewall.
setenv GLOBUS_TCP_PORT_RANGE_STATE_FILE <location on file system>

# Set GLOBUS_TCP_SOURCE_RANGE_STATE_FILE to the location where Globus should record
# TCP port usage for inbound connections in case of a stateful firewall.
setenv GLOBUS_TCP_SOURCE_RANGE_STATE_FILE <location on file system>

All firewalls (host-based and stateful)

These are configurations you can use to set environment variables such that services and users know about the firewall configuration. A future version of osg-configure will do these for you.

Globus gatekeeper (for the OSG Compute Element)

Edit /etc/sysconfig/globus-gatekeeper to add:

# Set GLOBUS_TCP_PORT_RANGE to define communication ports for outbound connections.
export GLOBUS_TCP_PORT_RANGE=<begin port>,<end port>

# Set GLOBUS_TCP_SOURCE_RANGE to define communication ports for inbound connections.
export GLOBUS_TCP_SOURCE_RANGE=<begin port>,<end port>

Globus GridFTP:

Edit /var/lib/osg/globus-firewall to add:

# Set GLOBUS_TCP_PORT_RANGE to define communication ports for outbound connections.
export GLOBUS_TCP_PORT_RANGE=<begin port>,<end port>

# Set GLOBUS_TCP_SOURCE_RANGE to define communication ports for inbound connections.
export GLOBUS_TCP_SOURCE_RANGE=<begin port>,<end port>

User environments

You can tell configure the user environment by editing two files:

/etc/profile.d/globus_firewall.sh

# Set GLOBUS_TCP_PORT_RANGE to define communication ports for outbound connections.
export GLOBUS_TCP_PORT_RANGE=<begin port>,<end port>

# Set GLOBUS_TCP_SOURCE_RANGE to define communication ports for inbound connections.
export GLOBUS_TCP_SOURCE_RANGE=<begin port>,<end port>

/etc/profile.d/globus_firewall.csh

# Set GLOBUS_TCP_PORT_RANGE to define communication ports for outbound connections.
setenv GLOBUS_TCP_PORT_RANGE <begin port>,<end port>

# Set GLOBUS_TCP_SOURCE_RANGE to define communication ports for inbound connections.
setenv GLOBUS_TCP_SOURCE_RANGE <begin port>,<end port>

Testing and Monitoring

Use telnet to check whether the ports required are accessible on your Compute Element or Storage Element by trying to connect to the respective TCP port from the outside:

[user@ce ~]$ telnet <the FQDN of the CE or SE> <port number to check>

The port is accessible if you get a response. Otherwise telnet will hang or report a no route to host error.

If you configured iptables to log events, you should be able to see blocked events by inspecting /var/log/messages.

References

  1. Globus Firewall Settings (old document)
  2. Globus Firewall HowTo
  3. Globus document Globus-Firewall-Requirements-9.pdf
  4. HTCondor Firewall Settings
  5. Wikipedia Article on Firewalls
  6. Wikipedia Article on IPTables
  7. Wikipedia Article on Stateful Firewalls

Comments

Topic attachments
I Attachment Action Size Date Who Comment
pdfpdf Globus-Firewall-Requirements-9.pdf manage 296.1 K 27 Nov 2012 - 16:02 MarcoMambelli  
pngpng firewall-client-12p.png manage 13.1 K 01 Feb 2012 - 14:25 MarcoMambelli  
pngpng firewall-client.png manage 32.9 K 01 Feb 2012 - 08:09 MarcoMambelli  
pngpng firewall-net1-12p.png manage 53.0 K 01 Feb 2012 - 14:25 MarcoMambelli  
pngpng firewall-net1.png manage 127.7 K 01 Feb 2012 - 08:10 MarcoMambelli  
pngpng firewall-server-12p.png manage 14.8 K 01 Feb 2012 - 14:24 MarcoMambelli  
pngpng firewall-server.png manage 37.3 K 01 Feb 2012 - 14:25 MarcoMambelli  
Topic revision: r48 - 06 Dec 2016 - 18:12:40 - KyleGross
Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..