Troubleshooting glexec and lcmaps

1 About this document

This document will help you troubleshoot problems glexec. We assume that glexec has already been installed. See also other documents recommended in the Reference section below.

This document follows the general OSG documentation conventions:

on on

2 How to get Help?

To get assistance please use our Help Procedure.

3 Glexec: The big picture

3.1 What software is used?

While we casually refer to glexec, you need to be aware of three different pieces:

  1. glexec: Glexec is a program that you can think of a "GSI sudo". It will run a program as another user, but first it will authenticate and authorize the submitting user based on their X.509 credential.
  2. lcmaps: lcmaps is a framework for authorizing a user and mapping their X.509 credential to a local user name. Glexec doesn't perform authorization itself, but instead uses lcmaps.
  3. lcmaps plugins: By itself, lcmaps does very little. Instead it relies on a set of plugins to do all of the work. You need to be aware of these plugins because they show up in the lcmaps configuration. When people refer to "lcmaps", they really mean "lcmaps plus plugins".

3.2 Configuring policies

Lcmaps can be used by different pieces of software. In OSG, we currently (mid 2012) use lcmaps not only for glexec but also for Globus GRAM (for a Compute Element Gatekeeper) and GridFTP (for a Storage Element) when we use GUMS. (Normally GRAM and GridFTP don't bother with lcmaps when using a grid-mapfile, though they could.) Because lcmaps can be used in different contexts, the /etc/lcmaps.db file describes how to use lcmaps in each of those contexts. On a single machine, you probably only use it one way, but our default configuration file is set up for all our possible uses.

The /etc/lcmaps.db file is broken into two sections:

  1. Information about each of the plugins. Normally you don't need to edit these except to set the GUMS hostname and to enable the glexectracking plugin. (It's disabled by default because it won't be installed when using lcmaps with just Globus and not glexec)
  2. Policies. Our default configuration ships with two policies:
    1. authorize_only: Used by Globus-based services.
    2. glexec: Used by glexec. The details of the policy are uncommented because you have to make choices when you install it.

4 Common Glexec problems

You can try out glexec with something like the following:

[user@client ~]$ voms-proxy-init -voms yourvo:/yourvo
[user@client ~]$ export GLEXEC_CLIENT_CERT=/tmp/x509up_uNNN
[user@client ~]$ /usr/sbin/glexec /usr/bin/id

If your lcmaps.db is set up to not use a host certificate as described in GlexecPilotCert, you should also set

[user@client ~]$ export X509_USER_PROXY=tmp/x509up_uNNN
before running glexec.

4.1 Didn't enable glexec policy

[user@client ~]$ /usr/sbin/glexec /usr/bin/id
[gLExec]:  LCMAPS failed.
           The reason can be found in the syslog.
[user@client ~]$ echo $?
203

The lcmaps error in /var/log/messages indicates a configuration problem:

May 23 08:20:09 fermicloud037 glexec[2823]: Trying to read /etc/glexec.conf as 13975(glexec)/9815(glexec) 
May 23 08:20:09 fermicloud037 glexec[2823]: lcmaps: /etc/lcmaps.db:158: [warning] expecting rule definitions. 
May 23 08:20:09 fermicloud037 glexec[2823]: lcmaps: /etc/lcmaps.db:158: [warning] no rules specified for policy: 'glexec' at line 141.
May 23 08:20:09 fermicloud037 glexec[2823]: lcmaps: LCMAPS failed to do mapping and return account information 
May 23 08:20:09 fermicloud037 glexec[2823]:   LCMAPS failed. 

This probably indicates the you didn't edit /etc/lcmaps.db after installing. By default, that file does not have a policy enabled: you have to select one by uncommenting it in the file. The glexec policy portion of the lcmaps.db file looks like this when it's installed: note that all the lines begin with a #, indicating that they are commented out:

# Mapping policy: glexec
# Purpose:        Used for glexec on the worker nodes.
#
glexec:

## If you use glexec, pick an appropriate policy from the ones below.
## Make sure that only one policy is uncommented.

## Policy 1: GUMS but not SAZ (most common)
#verifyproxy -> gumsclient
#gumsclient -> glexectracking

## Policy 2: GUMS & SAZ
#verifyproxy -> sazclient
#sazclient -> gumsclient
#gumsclient -> glexectracking

## Policy 3: grid-mapfile
#verifyproxy -> gridmapfile
#gridmapfile -> glexectracking

Most people will select the first policy. If you do, it will look like this:

# Mapping policy: glexec
# Purpose:        Used for glexec on the worker nodes.
#
glexec:
verifyproxy -> gumsclient
gumsclient -> glexectracking

4.2 Didn't enable glexectracking

[user@client ~]$ /usr/sbin/glexec /usr/bin/id
NO OUTPUT
[user@client ~]$  echo $?
202

The lcmaps error in /var/log/messages indicates a configuration problem:

May 23 08:21:40 fermicloud037 glexec[2844]: Trying to read /etc/glexec.conf as 13975(glexec)/9815(glexec) 
May 23 08:21:40 fermicloud037 glexec[2844]: lcmaps: lcmaps.mod-PluginInit(): plugin glexectracking not found (arguments: )
May 23 08:21:40 fermicloud037 glexec[2844]: lcmaps: lcmaps.mod-lcmaps_startPluginManager(): error initializing plugin: glexectracking 
May 23 08:21:40 fermicloud037 glexec[2844]: lcmaps: lcmaps_init() error: could not start plugin manager 
May 23 08:21:40 fermicloud037 glexec[2844]:   Initialisation of LCMAPS failed. 

You probably requested the use of the glexec tracking plugin in /etc/lcmaps.db, since that's what we recommend:

glexec:
verifyproxy -> gumsclient
gumsclient -> glexectracking

But you probably didn't uncomment glexectracking in /etc/lcmaps.db to tell lcmaps where to find it.

# Uncomment the first two lines in order to enable glexec tracking
#glexectracking = "lcmaps_glexec_tracking.mod"
#         "-exec /usr/sbin/glexec_monitor"

Just remove those hash marks to enable glexectracking:

# Uncomment the first two lines in order to enable glexec tracking
glexectracking = "lcmaps_glexec_tracking.mod"
         "-exec /usr/sbin/glexec_monitor"

4.3 User doesn't exist

[user@client ~]$ /usr/sbin/glexec /usr/bin/id
[gLExec]:  LCMAPS failed.
           The reason can be found in the syslog.
[user@client ~]$ echo $?
203

The lcmaps error in /var/log/messages indicates the user doesn't exist:

May 23 08:22:46 fermicloud037 glexec[2860]: Trying to read /etc/glexec.conf as 13975(glexec)/9815(glexec) 
May 23 08:22:46 fermicloud037 glexec[2860]: lcmaps: Username_handler: Error: Couldn't find the username 'engage' in the password file.

Make the user account to fix this problem.

4.4 The user doesn't have a proxy

[user@client ~]$ /usr/sbin/glexec /usr/bin/id
[gLExec]:  Failed to open $GLEXEC_CLIENT_CERT=/tmp/x509up_u11500 or $GLEXEC_SOURCE_PROXY=(NULL).
[user@client ~]$ echo $?
201

Make a proxy with voms-proxy-init to solve this problem.

4.5 User isn't authorized

[user@client ~]$ /usr/sbin/glexec /usr/bin/id
[gLExec]:  LCMAPS failed.
           The reason can be found in the syslog.

The lcmaps error in /var/log/messages indicates the user isn't authorized by your GUMS server:

May 23 08:55:05 fermicloud037 glexec[5981]: lcmaps: LCMAPS failed to do mapping and return account information

4.6 Missing hostkey

glexec requires a host certificate on each worker node in order to talk to GUMS. When you don't have a hostcert, you get an error message like this in /var/log/messages

(output was word-wrapped for clarity)

May 23 08:36:55 fermicloud037 glexec[3097]: Syslog has been reopened with new facility 'LOG_LOCAL1' 
May 23 08:36:55 fermicloud037 glexec[3097]: lcmaps: setup_client_ctx: Error loading private key from file: '/etc/grid-security/hostkey.pem'. 
    OpenSSL reason: No such file or directory.
May 23 08:36:55 fermicloud037 glexec[3097]: lcmaps: SSL_client_connect: Error: can't create SSL handle out of CTX structure 
May 23 08:36:55 fermicloud037 glexec[3097]: lcmaps: ssl_io_connect(): Failure in SSL layer setup and connection! 

4.7 Can't connect to GUMS server

Here's an example error from /var/log/messages when you can't connect to the GUMS server. Make sure that you've used the proper hostname for the GUMS server in /etc/lcmaps.db and the GUMS server is running.

(output was word-wrapped for clarity)

May 23 08:38:22 fermicloud037 glexec[3121]: lcmaps: post_connection_check: Error: found one or more DNS entries in the subjectAltName but none match the host 'cloudgums1.fnal.gov'. 
May 23 08:38:22 fermicloud037 glexec[3121]: lcmaps: ssl_io_connect(): Error with peer certificate after the full handshake: application verification failure 
May 23 08:38:22 fermicloud037 glexec[3121]: lcmaps: xacmlqueryscas(): XACML: 
    Interaction failed: TCP/IP, SSL or SOAP Error with endpoint: 
    "https://cloudgums1.fnal.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort" 

4.8 A successful authorization

When glexec works, it will look something like this:

(output was word-wrapped for clarity)

May 23 08:24:38 fermicloud037 glexec[2913]: Trying to read /etc/glexec.conf as 13975(glexec)/9815(glexec) 
May 23 08:24:38 fermicloud037 glexec[2913]: lcmaps: lcmaps_plugin_scas_client-plugin_run(): 
    User "/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511" 
    with first FQAN "/Engage/Role=NULL/Capability=NULL" 
    Permitted at endpoint "https://cloudgums.fnal.gov:8443/gums/services/GUMSXACMLAuthorizationServicePort" 
May 23 08:24:38 fermicloud037 glexec.mon[2914#2913]: Started, target uid 46661
May 23 08:24:38 fermicloud037 glexec.mon[2914#2913]: Used DN: "/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511"
May 23 08:24:38 fermicloud037 glexec.mon[2914#2913]: Used VO: "Engage" Issuer: "/DC=org/DC=doegrids/OU=Services/CN=osg-engage.renci.org"
May 23 08:24:38 fermicloud037 glexec.mon[2914#2913]: Used FQAN: "/Engage/Role=NULL/Capability=NULL"
May 23 08:24:39 fermicloud037 glexec[2913]: lcmaps: LCMAPS CRED FINAL: DN:"/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511"
    ->mapped uid:'46661',pgid:'46661',sgid:'46661',sgid:'65000' 
May 23 08:24:39 fermicloud037 glexec[2913]: Ignoring SIGCHLD from unknown finished child 2914. 
May 23 08:24:39 fermicloud037 glexec[2913]: real_user(alainroy),real_group(cdf)
    ->mapped_user(engage),mapped_group(engage) cwd: "/home/engage" cmd: "/usr/bin/id" (="/usr/bin/id")

4.9 Other possible problems.

If you have upgraded glexec with yum, you might need to merge changes from your old configuration file into the new configuration file.

Existing, old configuration file: /etc/glexec.conf
New configuration file template: /etc/glexec.conf.rpmnew

Similarly, if you have upgraded lcmaps with yum, you might need to merge changes from your old configuration file into the new configuration file.

Existing, old configuration file: /etc/lcmaps.db
New configuration file template: /etc/lcmaps.db.rpmnew

Check to make sure that you are running a recent version of glexec and lcmaps.

5 Appendix: Important glexec files

This document cannot cover all the errors you might experience. If you need to look for more data, you can look at log files for the various services on your CE.

File Purpose
/usr/sbin/glexec The glexec program
/etc/glexec.conf The glexec configuration
/etc/lcmaps.db The lcmaps configuration

The most common RPMs you will see are:

RPM Purpose
glexec The glexec RPM
lcmaps The base lcmaps RPM
lcmaps-plugins-gums-client The lcmaps plugin that communicates with GUMS
lcmaps-plugins-verify-proxy The lcmaps plugin that verifies the user's proxy is good
lcmaps-plugins-glexec-tracking The lcmaps plugin that kills processes started by glexec

6 References

7 Comments

Topic revision: r12 - 06 Dec 2016 - 18:12:45 - KyleGross
Hello, TWikiGuest!
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..