Guide to using OSG for users of the TGYRO application.


This guide is specifically aimed at enabling TGYRO users to take advantage of the Open Science Grid. It may look daunting at first, but a large fraction of these instructions only need to be followed once; yet more need to be done only once per year. There are two main sections: Authorization and authentication dealing with how to get the required access to run via the OSG, and Running TGYRO jobs on the OSG which tells you what you need to know to submit jobs to OSG sites and obtain the results.

Authorization and authentication.

Perhaps the biggest difference from the methods for running TGYRO with which you are familiar is that one logs onto one machine, in order to submit jobs to any one of several sites. The authentication and authorization requirements of those sites are generally handled using X.509 certificates. The ones used are so-called, "extended attribute" certificates known as, "VOMS proxies." These extended attributes encode information about your, "Virtual Organization" and, particular scientific purpose to enable sites to decide whether to allow you access and how to prioritize your usage with respect to everyone else.

Obtaining an X.509 certificate (once only).

The first thing one needs to do is obtain an X.509 certificate. If you do not already have an X.509 certificate of your own, you should obtain one from the DOE. In order to do this you should use a browser, preferably Mozilla Firefox.

Ensure your browser recognizes and accepts DOE certificates.

Apply for a personal certificate

To apply for a personal certificate, use a web browser and go to:

Select "Request New Certificate" from the menu in the upper left.. For Virtual Organization name, select ' Engage '. Please include the name of your Engage contact in the comments section. If you require help, please contact the Engage team at engage-team@opensciencegrid dot org

Retrieve your personal certificate

When your certificate has been successfully issued, you will receive an email that contains a link to a page containing all your certificate information. Open that page in your browser and click on Import Your Certificate button at the bottom of that page.

Export your certificate for use in job submission.

See the instructions on the DOEGrids certificate page.

Join the Engage VO (once only).

Once you have your certificate, you can join the Engage Virtual Organization (VO). Go to:

The browser should present your certificate to the server, and you should see something like this on the bottom of the page (but with your name):

DN: /DC=com/DC=DigiCert-Grid/O=Open Science Grid/OU=People/CN=JANE DOE 72645

Fill out and submit the form.

Specify your VO representative as, "Christopher H. Green (Engage)."

Ensure that you specify:

<your-institute> (TGYRO)
as your affiliation to ensure that your application gets routed appropriately.

Be sure to follow immediately the instructions in the "Phase I" email you will receive in order for your application to progress.

You and/or your institute's TGYRO / ITER representative may receive further inquiries, but you should normally receive an email notification that you have been admitted to the Engage VO within one business day. Follow the instructions therein to finalize your admittance to the VO.

Obtain access to the central Engage VO submit node.

Visit the RENCI account request page and fill in the following information:

  • RENCI contact: John McGee
  • RENCI contact e-email:
  • Please select "Don't need one" regarding: If you need to be added to the gridmap file please select the appropriate option.
  • Please give a short description of why you need this account: OSG via Chris Green.

Renewing your DOEGrids certificate (once per year only).

A few weeks before your certificate arrives you will start receiving emails warning you of the impending expiration of your certificate. In order to renew your certificate at any time before it expires, do the following:

  1. Ensure you have your current certificate in your browser. For greatest ease use the same browser from which you requested the certificate in the first place; otherwise import from the backup you made (and kept, right?)
  2. Visit and select the "Certificate Renewal" link from the left sidebar.
  3. Follow the instructions. Your certificate will be automatically renewed and imported into your browser.
  4. Follow the instructions above.

If your certificate has expired, you will need to request a new one. In the comments field, request that your certificate be issued with the same DN (including "random" number) as your existing one, and paste the DN in the comments field also. This can be obtained either from your browser or certificate handling utility or by executing the command from your submit node:

openssl x509 -noout -in $HOME/.globus/usercert.pem -subject

Re-signing the VO Usage Policy (once per year only).

Once per year, you should receive an email directing you to re-sign the Acceptable Use Policy, or AUP. Read and agree to abide by the AUP in order to continue membership of the Engage VO for another year.

Obtain VOMS credentials for job submission (prior to job submission but not more often than once per 24h).

Note that you will have to have your certificate exported as mentioned above, and placed in ~/.globus/ on the submit host. A proxy is a time limited version of your certificate. The proxy is what is actually presented to the remote site during authentication. To create a proxy, use the voms-proxy-init command:

[rynge@nantahala ~]$ voms-proxy-init -voms Engage:/Engage/Fusion/TGYRO -valid 72:00
Cannot find file or dir: $prefix/etc/vomses
Your identity: /DC=org/DC=doegrids/OU=People/CN=Mats Rynge 722233
Enter GRID pass phrase:
Creating temporary proxy ...................................... Done
Contacting [/DC=org/DC=doegrids/OU=Services/] "Engage" Done

Creating proxy ............................................ Done
Your proxy is valid until Thu Apr  5 10:29:08 2007

Note that you can specify how long the proxy should be valid for. It should be long enough for the job run to finish, but should not be longer than 72 hours. You might see some warnings in the output of voms-proxy-init. Do not worry about them as long as you get a proxy at the end. You can check the proxy with voms-proxy-info:

[rynge@nantahala ~]$ voms-proxy-info
WARNING: Unable to verify signature! Server certificate possibly not installed.
Error: Cannot find certificate of AC issuer for vo Engage
subject   : /DC=org/DC=doegrids/OU=People/CN=Mats Rynge 722233/CN=proxy
issuer    : /DC=org/DC=doegrids/OU=People/CN=Mats Rynge 722233
identity  : /DC=org/DC=doegrids/OU=People/CN=Mats Rynge 722233
type      : proxy
strength  : 512 bits
path      : /tmp/x509up_u1031
timeleft  : 23:20:17

Site-specific authorization steps (once only per site).

Some sites require special steps to be taken to obtain access. These are listed below, along with the specific actions required.

NERSC sites.

You must be an authorized user of NERSC resources. If you are not, talk to your project management and follow your group's procedures to obtain access, and in particular, NIM login abilities. Assuming this is done, access the NIM site and look for the, "Grid Certificates" tab on the right of the main frame. Use the link provided to add your certificate to the list. You will need the information obtained by executing the following command on your grid submission node (the one onto which you placed your DOE certificate):

openssl x509 -noout -in $HOME/.globus/usercert.pem -subject

Running TGYRO jobs on the OSG.

Before attempting to run jobs on the OSG, ensure first that you are able to log on to the current Engage submit host at RENCI, NC, and that you are able to obtain the correct user proxy from there. Then:

  1. Ensure you have the correct environment for TGYRO job submission:
    . ~greenc/tgyro-suite/
    source ~greenc/tgyro-suite/setup.csh
    as appropriate.
  2. Ensure you have a local top-level work directory (eg ~/mpi-work/tgyro).
  3. Generate your work area, either by importing or copying an older directory or by generating it from a template, eg:
    tgyro -g stest_1 -p ~/mpi-work/tgyro
  4. Submit your job as normal, eg:
    tgyro_bat -e stest_1 -n 4 -p ~/mpi-work/tgyro
    Answer the questions as appropriate.
  5. Monitor the progress of your job with condor_q <user> or condor_q <job#>.
  6. Upon completion of the job, expand the results archive, output_files.tar.bz2.

Submission of TGYRO jobs is still being improved, so please bookmark this section and check back regularly for news of any changes.

If you are interested, here are some technical details of how the TGYRO system was adapted to run on the OSG. -- ChrisGreen - 18 Sep 2009

Topic revision: r8 - 25 Oct 2011 - 17:22:34 - MatsRynge
Hello, TWikiGuest


TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..