Data Management Exercises (Part I)

Introduction

The storage and data management are an essential part of grid computing. Many applications running on the OSG sites are data-intensive and require high-throughput. The middleware provided by the OSG can manage not only jobs but also transfers of input and output files associated with a job, as well as pre-staging of the data before launching jobs. The goals of these exercises are:

  • to be able to pre-stage file using globus-url-copy or srm-copy commands
  • to familiarize yourself with various SRM commands that allow data management and movement on a Storage Element
  • to submit Condor-G job that accesses file that is stored at a SE
  • to submit Condor-G job that first try to get file from the web using Squid

Customize this Document

warning Please change your user name and click on the Customize button!

VO
Host Name
Domain Name
Login Name
Exercise Path
Gatekeeper Name
Batch System
OSG_DATA
SURL
 

Exercises

Prerequisite

  • Login on submission node
    ssh user@submit03.ncc.unesp.br
    
  • Obtain proxy certificate, if you have not done so already
    voms-proxy-init -voms osgedu:/osgedu
    Enter GRID pass phrase: 
    
  • Make a directory for this exercise
    cd
    mkdir -p osg_school/dm_part_1
    cd osg_school/dm_part_1
    

Classic Storage

A Classic Storage is represented by a stand-alone GridFTP server installed on a Compute Element. It allows access to several specific data directories available on all OSG sites. The storage space allocated under $OSG_DATA directory is intended as the space for applications to write input and output of relatively small data files which are kept around for longer than the job which created them.

In order to upload data to $OSG_DATA you will need to use globus-url-copy command. The basic syntax for globus-url-copy is:

globus-url-copy [optional command line switches] Source_URL Destination_URL

You will need to use the following URL prefixes to specify local file and file on the remote node:

  • file:// (on a local machine only)
  • gsiftp://

  • What is the path to OSG_DATA?
    globus-job-run ce.sp.grid.unesp.br:/jobmanager-fork /usr/bin/env|grep OSG_DATA
    OSG_DATA=/osg/data
    

  • Let's create your directory there
    globus-job-run ce.sp.grid.unesp.br:/jobmanager-fork /bin/mkdir /osg/data/osgedu/${USER}
    
  • Get a copy of our data file and worker script for these exercises
    cp /opt/workshop/dm_exercises/part1/Complete-Prose-Works-by-Walt-Whitman.txt .
    cp /opt/workshop/dm_exercises/part1/word_counter .
    
    The data file size is 1.5 MB and it contains works by Walt Whitman. (The file is originally from this site, for more info see information about Gutenberg Project ). Let's Use globus-url-copy to upload the file to your directory under /osg/data/osgedu:
     
    globus-url-copy file:///home/${USER}/osg_school/dm_part_1/Complete-Prose-Works-by-Walt-Whitman.txt gsiftp://ce.sp.grid.unesp.br:2811//osg/data/osgedu/${USER}/Complete-Prose-Works-by-Walt-Whitman.txt
    globus-job-run ce.sp.grid.unesp.br:/jobmanager-fork /bin/ls -l /osg/data/osgedu/${USER}
    total 1416
    -rw-r--r--  1 osgedu users 1445762 Jul 12 13:17 Complete-Prose-Works-by-Walt-Whitman.txt
    
  • The word_counter script calculates number of occurrences of a specified word in a text file. The script looks like this
    
    more word_counter  More...  Close    
    #!/usr/bin/python
    import sys
    import re
    ####################################################################################
    #                                                                                                                                                                   #
    # word_counter: Calculates number of occurrences of a specified word in a text file                                   #
    #               output goes to a log file, is not specified to stdout                                                                       #
    ####################################################################################
    
    class Text:
            def __init__(self,file,log):
                    self.text=[]
                    self.log=log
                    self.readFile(file)
            def readFile(self,file):
                    text=""
                    try:
                            fd=open(file,'r')
                            for  line in fd.readlines():
                                    text="%s %s" % (text, re.sub(r'\W+',' ',line[:-1].lower()))
                            self.text=re.split('\W+',text)
                            self.text.sort()
                    except:
                            log.write("Failed to read text from file %s: %s %s\n" % (file,sys.exc_type,sys.exc_value))
                            raise "TextError"
    
    
            def getCount(self,word):
                    counter=0
                    try:
                            indx=self.text.index(word)
                            for item in self.text[indx:]:
                                    if word==item:
                                            counter=counter+1
                                    else:
                                            break
                    except:
                            log.write("Failed to find word %s: %s %s\n" % (word,sys.exc_type,sys.exc_value))
                    return counter
    
    if __name__=="__main__":
            from optparse import OptionParser
            log=sys.stdout
            retVal=0
            usage="Usage:%s [options] text_file word\n" % (sys.argv[0])
            parser = OptionParser(usage=usage)
            parser.add_option("-o", "--output",dest="logfile",help="script logfile")
            (options, args) = parser.parse_args()
            if options.logfile != None:
                    try:
                            log=open(options.logfile,'a')
                            log.write("Open logfile %s\n" % (options.logfile))
                    except:
                            log=sys.stdout
                            log.write("Failed to open logfile %s : %s %s\n" % (options.logfile,sys.exc_type,sys.exc_value))
            if len(args) != 2:
                    log.write(usage)
                    retVal=1
            else:
                    try:
                            text=Text(sys.argv[1],log)
                            log.write("Number of word \"%s\" encountered in %s is %s\n" % (sys.argv[2],sys.argv[1],text.getCount(sys.argv[2])))
                    except:
                            log.write("Failed to parse text : %s %s\n" % (sys.exc_type,sys.exc_value))
                            retVal=2
            log.close()
            sys.exit(retVal)
      
    
    
    Let's execute this script to verify that it is working properly:
    ./word_counter Complete-Prose-Works-by-Walt-Whitman.txt leaves  
    Number of word "leaves" encountered in Complete-Prose-Works-by-Walt-Whitman.txt is 74
    

  • Create simple Condor-G job submission files that access Whitman's prose file from various sources:
    1. Transfer data with condor job. First, create a submission file test1.submit
      universe=grid
      grid_resource = gt2 ce.sp.grid.unesp.br/jobmanager-condor
      Executable=word_counter
      Log = test1.$(Cluster).$(Process).log
      Output = test1.$(Cluster).$(Process).out
      Error = test1.$(Cluster).$(Process).err
      WhenToTransferOutput = ON_EXIT_OR_EVICT
      transfer_input_files = Complete-Prose-Works-by-Walt-Whitman.txt
      transfer_output_files = wordcounter_results.test1.$(Cluster).$(Process)
      Arguments =  Complete-Prose-Works-by-Walt-Whitman.txt leaves --output=wordcounter_results.test1.$(Cluster).$(Process)
      queue
      
      Submit the job:
      condor_submit test1.submit
      Submitting job(s).
      Logging submit event(s).
      1 job(s) submitted to cluster 13.
      
      Check status of the job. Verify the content of the job log file (wordcounter_results.test1.13.0) after it is done. It should contain something similar to that:
      $ cat wordcounter_results.test1.13.0
      Open logfile wordcounter_results.test1.13.0
      Number of word "leaves" encountered in Complete-Prose-Works-by-Walt-Whitman.txt is 74
      
    2. Now, we can submit a job that is using file from /osg/data/osgedu/${USER} directory that we have uploaded before. Create test2.submit
      universe=grid
      grid_resource = gt2 ce.sp.grid.unesp.br/jobmanager-condor
      Executable=word_counter
      Log = test2.$(Cluster).$(Process).log
      Output = test2.$(Cluster).$(Process).out
      Error = test2.$(Cluster).$(Process).err
      WhenToTransferOutput = ON_EXIT_OR_EVICT
      transfer_output_files = wordcounter_results.test2.$(Cluster).$(Process)
      Arguments =  /osg/data/osgedu/user/Complete-Prose-Works-by-Walt-Whitman.txt grass  --output=wordcounter_results.test2.$(Cluster).$(Process)
      queue
      
      Submit job:
      condor_submit test2.submit
      
      Check the status of the job. Verify the contents of the job output file after it is done.

You can see we used too different methods. The first is more convenient because you didn't need to pre-stage the data, but if you need to run many different jobs against the same data set, you should prefer to pre-stage the data.

On your own

  • Create Condor-G submission file that access BLAST databases from $OSG_DATA(/osg/data/osgedu/blast_dbs) and run BLAST application ($OSG_APP/hcc/blast) installed under $OSG_APP and store output back on $OSG_DATA.
  • Make a two-node DAG that does the data transfer in one node and the analysis in the second node. Emphasize that if you had more than one analysis job, this could be an efficient way to process data.
  • GridFTP allows to do third-party transfer. Try to initiate a data transfer from a client running on vtb-itb to transfer data from Wisconsin to Nebraska.
  • You can speed GridFTP transfers: try to use bigger tcp window, memory buffer and parallel streams.

Storage Element

An OSG site may offer a Storage Element (SE) - a physical storage system where data is stored and managed according to authorization policies. A SE could be represented by a physical file system, disk caches or hierarchical mass storage system. A Storage Resource Manager (SRM) is a component that is usually associated with a Storage Element. A SRM optimizes the use of a SE by providing file caching, staging of files from hierarchical storage, and/or managing the space by using space reservation and lifetime of the file. Storage Elements manage storage and enforce authorization policies on who is allowed to create, delete and access physical files. See this document for details. All OSG Storage Elements support the gsiftp protocol and the full or limited implementation of Storage Resource Manager (SRM) specification. In some cases a Storage Element and the shared file system may use the same underlying distributed file system, providing local access from the worker nodes and external access through the gsiftp protocol, or the SRM specification.

During this exercise we will pre-stage data on the specific SE and then run a job on a worker node that accesses the file directly.

What you have to know before you start:

  • Storage url (surl) of Storage Elements available for osgedu VO members:
    1. Nebraska SE end point: srm://red-srm1.unl.edu:8443/srm/v2/server?SFN=/mnt/hadoop/user/osgedu (BeStMan-gateway/Hadoop DFS; data available from the worker nodes with the path /mnt/hadoop/user/osgedu)
    2. OSG_EDU SE end point: srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/

There are many implementations of srm client. You can choose the one you like the most. Examples that are shown here are using srm-lbnl-client.

First steps

  • Verify that you can access storage
    srm-ping srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store
      More...  Close    
    srm-ping   2.2.1.3.12  Tue Apr 27 13:13:24 PDT 2010
    BeStMan and SRM-Clients Copyright(c) 2007-2010,
    Lawrence Berkeley National Laboratory. All rights reserved.
    Support at SRM@LBL.GOV and documents at http://sdm.lbl.gov/bestman
    SRM-CLIENT: Connecting to serviceurl httpg://se.grid.unesp.br:8443/srm/v2/server
    
    SRM-PING: Mon Dec 06 19:43:36 BRST 2010  Calling SrmPing Request...
    versionInfo=v2.2
    
    Extra information (Key=Value)
    backend_type=BeStMan
    backend_version=2.2.1.3.13
    backend_build_date=2010-04-28T16:55:52.000Z 
    gsiftpTxfServers[0]=gsiftp://dts01.grid.unesp.br
    gsiftpTxfServers[1]=gsiftp://dts02.grid.unesp.br
    gsiftpTxfServers[2]=gsiftp://dts03.grid.unesp.br
    gsiftpTxfServers[3]=gsiftp://dts04.grid.unesp.br
    GatewayMode=Enabled
    clientDN=/DC=org/DC=doegrids/OU=People/CN=Tanya Levshina 508821
    gumsIDMapped=osgedu
     
    

  • Verify that you can upload file. First, create a directory
    srm-mkdir  srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/${USER}
      More...  Close    
    srm-mkdir   2.2.1.3.12  Tue Apr 27 13:13:24 PDT 2010
    BeStMan and SRM-Clients Copyright(c) 2007-2010,
    Lawrence Berkeley National Laboratory. All rights reserved.
    Support at SRM@LBL.GOV and documents at http://sdm.lbl.gov/bestman
    SRM-CLIENT: Connecting to serviceurl httpg://se.grid.unesp.br:8443/srm/v2/server
    
    SRM-DIR: Mon Dec 06 19:46:22 BRST 2010 Calling SrmMkdir
    SRM-DIR: DirectoryPath(0)=srm://se.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/tanya
            status=SRM_SUCCESS
            explanation=null
     
    
    Copy Whitman's prose file to this SE directory:
    srm-copy file:///${PWD}/Complete-Prose-Works-by-Walt-Whitman.txt   srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/${USER}/Complete-Prose-Works-by-Walt-Whitman.txt
      More...  Close    
    srm-copy   2.2.1.3.12  Tue Apr 27 13:13:24 PDT 2010
    BeStMan and SRM-Clients Copyright(c) 2007-2010,
    Lawrence Berkeley National Laboratory. All rights reserved.
    Support at SRM@LBL.GOV and documents at http://sdm.lbl.gov/bestman
    SRM-CLIENT: Mon Dec 06 19:47:21 BRST 2010 Connecting to httpg://se.grid.unesp.br:8443/srm/v2/server
    
    SRM-CLIENT: Mon Dec 06 19:47:22 BRST 2010 Calling SrmPrepareToPutRequest now ...
    request.token=put:33052
    Request.status=SRM_SUCCESS
    explanation=null
    
    SRM-CLIENT: RequestFileStatus for SURL=file:////home/tanya/osg_school/dm_part_1/Complete-Prose-Works-by-Walt-Whitman.txt is Ready.
    SRM-CLIENT: received TURL=gsiftp://dts01.grid.unesp.br//store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    
    SRM-CLIENT: Mon Dec 06 19:47:26 BRST 2010 start file transfer
    SRM-CLIENT:Source=file:////home/tanya/osg_school/dm_part_1/Complete-Prose-Works-by-Walt-Whitman.txt
    SRM-CLIENT:Target=gsiftp://dts01.grid.unesp.br//store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    
    SRM-CLIENT: Mon Dec 06 19:47:30 BRST 2010 end file transfer for file:////home/tanya/osg_school/dm_part_1/Complete-Prose-Works-by-Walt-Whitman.txt
    
    SRM-CLIENT: Mon Dec 06 19:47:30 BRST 2010 Calling putDone for srm://se.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    Result.status=SRM_SUCCESS
    Result.Explanation=null
    
    SRM-CLIENT: Request completed with success
    
    SRM-CLIENT: Printing text report now ...
    
    SRM-CLIENT*REQUESTTYPE=put
    SRM-CLIENT*TOTALFILES=1
    SRM-CLIENT*TOTAL_SUCCESS=1
    SRM-CLIENT*TOTAL_FAILED=0
    SRM-CLIENT*REQUEST_TOKEN=put:33052
    SRM-CLIENT*REQUEST_STATUS=SRM_SUCCESS
    SRM-CLIENT*SOURCEURL[0]=file:////home/tanya/osg_school/dm_part_1/Complete-Prose-Works-by-Walt-Whitman.txt
    SRM-CLIENT*TARGETURL[0]=srm://se.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    SRM-CLIENT*TRANSFERURL[0]=gsiftp://dts01.grid.unesp.br//store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    SRM-CLIENT*ACTUALSIZE[0]=0
    SRM-CLIENT*FILE_STATUS[0]=SRM_SUCCESS
    SRM-CLIENT*EXPLANATION[0]=SRM-CLIENT: PutDone is called successfully
    
     
    
  • Check that the file is actually there:
    srm-ls  srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/${USER}/Complete-Prose-Works-by-Walt-Whitman.txt
      More...  Close    
    srm-ls   2.2.1.3.12  Tue Apr 27 13:13:24 PDT 2010
    BeStMan and SRM-Clients Copyright(c) 2007-2010,
    Lawrence Berkeley National Laboratory. All rights reserved.
    Support at SRM@LBL.GOV and documents at http://sdm.lbl.gov/bestman
    SRM-CLIENT: Connecting to serviceurl httpg://se.grid.unesp.br:8443/srm/v2/server
    
    SRM-DIR: Mon Dec 06 19:48:26 BRST 2010 Calling srmLsRequest
    
    SRM-DIR: ..........................
            Status    : SRM_SUCCESS
            Explanation : null
            Request token=null
    
            SURL=/store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
            Bytes=1497975
            FileType=FILE
            StorageType=null
            Status=SRM_SUCCESS
            Explanation=Read from disk..
            OwnerPermission=null
            LifetimeLeft=null
            LifetimeAssigned=null
            CheckSumType=null
            CheckSumValue=null
            FileLocality=ONLINE
            OwnerPermission=null
            GroupPermission=null
            OtherPermission=null
            ArrayOfSpaceTokens=null
            RetentionPolicyInfo=null
            LastModificationTime=null
            CreatedAtTime=null
    
    SRM-DIR: Printing text report now ...
    SRM-CLIENT*REQUEST_STATUS=SRM_SUCCESS
    SRM-CLIENT*SURL=/store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
    SRM-CLIENT*BYTES=1497975
    SRM-CLIENT*FILETYPE=FILE
    SRM-CLIENT*FILE_STATUS=SRM_SUCCESS
    SRM-CLIENT*FILE_EXPLANATION=Read from disk..
    SRM-CLIENT*FILELOCALITY=ONLINE
     
    
    * Remove this file:
    srm-rm  srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/${USER}/Complete-Prose-Works-by-Walt-Whitman.txt
      More...  Close    
    srm-rm   2.2.1.3.12  Tue Apr 27 13:13:24 PDT 2010
    BeStMan and SRM-Clients Copyright(c) 2007-2010,
    Lawrence Berkeley National Laboratory. All rights reserved.
    Support at SRM@LBL.GOV and documents at http://sdm.lbl.gov/bestman
    SRM-CLIENT: Connecting to serviceurl httpg://se.grid.unesp.br:8443/srm/v2/server
    
    SRM-DIR: Mon Dec 06 19:49:01 BRST 2010 Calling SrmRmFile
    SRM-DIR: Total files to remove: 1
            status=SRM_SUCCESS
            explanation=null
            surl=srm://se.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/tanya/Complete-Prose-Works-by-Walt-Whitman.txt
            status=SRM_SUCCESS
            explanation=null
     
    

On your own

  • Do srm-copy between Nebraska and OSG-EDU third-party transfer
    srm-copy  srm://SOURCE:port/srm/v2/server\?SFN=/filepath   srm://TARGET:port/srm/managerv2\?SFN=/filepath   -3partycopy
    

Running job on the Grid with SE

Run a Condor-G job that downloads Whitman's prose file from SE to the $OSG_WN_TMP area on the worker node and execute the same word_counter python script. First, make sure that you have the text file upload to the storage. Copy Whitman's prose file to this SE directory:
srm-copy file:///${PWD}/Complete-Prose-Works-by-Walt-Whitman.txt   srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/${USER}/Complete-Prose-Works-by-Walt-Whitman.txt

We will need to modify submission file first. Create test3.submit file

universe=grid
grid_resource = gt2 ce.sp.grid.unesp.br/jobmanager-condor
Executable = test3.sh
Log = test3.$(Cluster).$(Process).log
Output = test3.$(Cluster).$(Process).out
Error = test3.$(Cluster).$(Process).err
transfer_executable = true
transfer_input_files = word_counter
WhenToTransferOutput = ON_EXIT_OR_EVICT
transfer_output_files = wordcouner_results.test3.$(Cluster).$(Process)
should_transfer_files   = YES
Arguments =  Complete-Prose-Works-by-Walt-Whitman.txt grass srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/user wordcouner_results.test3.$(Cluster).$(Process)
queue
Copy test3.sh script from /opt/workshop/dm_exercises/part1/test3.sh.The script test3.sh will look like:
more test3.sh   More...  Close     
#!/bin/bash
############################################################
#  test3 - download file from a particular Storage Element  #
#          execute word_counter script                      #
#          do cleanup                                       #
#############################################################
usage ()
{
     log "Usage: file_name word surl path logfile"
}
log ()
{
if [ x$logname == x ]
then
        echo $1
else
        echo $1 >>$logfile
fi
}
# test if we have four arguments on the command line
if [ $# -lt  3 ]
then
    usage
    exit 1
fi
file=$1
word=$2
surl=$3
logname=""
cwd=`pwd`
if [ $# -eq 4 ]
then
        logname=$4
        logfile=$cwd/$logname
        #just in case create a logfile
        touch $logfile
fi
retVal=0
log "Start test3.sh"
#setup grid enviroment
. $OSG_GRID/setup.sh
#create temporary directory
export TMPDIR=$OSG_WN_TMP
tmpDir=`mktemp -d -t osgedu.XXXXXX`
cd $tmpDir
log "Current directory is $tmpDir"
#move and change the permission of the uploaded script
mv $cwd/word_counter .
chmod 755 word_counter
log "Executing: srm-copy $surl/$file file:///$PWD/$file"
srm-copy $surl/$file file:///$PWD/$file
retVal=$?
log "Return value $retVal"
if [ $retVal -eq 0 ]
then
        #move word_counter script
        option=""
        if [ x$logname != x ]
        then
                option="--output=$logfile"
        fi
        log "Executing: $cwd/word_counter $file  $word $option"
        ./word_counter $file  $word --output=$logfile
        retVal=$?
        log "Return value $retVal"
fi
cd $OSG_WN_TMP
rm -rf $tmpDir
log "End test3.sh"
exit $retVal
 

Running job on the Grid with SQUID

Run a Condor-G job that downloads Whitman's prose file via Squid service if available otherwise copy the file from SE. Place this file into the $OSG_WN_TMP area on the worker node then execute the same word_counter python script.

We will need to modify just slightly the submission file test3.submit. Create test4.submit file (note that the only difference between these two submission files is the name of the executable)

universe=grid
grid_resource = gt2 ce.sp.grid.unesp.br/jobmanager-condor
Executable = test4.sh
Log = test4.$(Cluster).$(Process).log
Output = test4.$(Cluster).$(Process).out
Error = test4.$(Cluster).$(Process).err
transfer_executable = true
transfer_input_files = word_counter
WhenToTransferOutput = ON_EXIT_OR_EVICT
transfer_output_files = wordcounter_results.test4.$(Cluster).$(Process)
should_transfer_files   = YES
Arguments =  Complete-Prose-Works-by-Walt-Whitman.txt grass srm://se.sp.grid.unesp.br:8443/srm/v2/server?SFN=/store/osgedu/user wordcounter_results.test4.$(Cluster).$(Process)
queue
Copy test4.sh script from /opt/workshop/dm_exercises/part1/test4.sh.The script test4.sh will look like:
more test4.sh   More...  Close     
#!/bin/bash
#############################################################
#  test4 - download file from a Squid server if exists or   #
# copy file from aparticular Storage Element                #
#          execute word_counter script                      #
#          do cleanup                                       #
#############################################################
usage ()
{
     log "Usage: file_name word surl path logfile"
}
log ()
{
if [ x$logname == x ]
then
        echo $1
else
        echo $1 >>$logfile
fi
}
# test if we have four arguments on the command line
if [ $# -lt  3 ]
then
    usage
    exit 1
fi
file=$1
word=$2
surl=$3
logname=""
cwd=`pwd`
if [ $# -eq 4 ]
then
        logname=$4
        logfile=$cwd/$logname
        #just in case create a logfile
        touch $logfile
fi
retVal=0
log "Start test4.sh"
#setup grid enviroment
. $OSG_GRID/setup.sh
#create temporary directory
export TMPDIR=$OSG_WN_TMP
tmpDir=`mktemp -d -t osgedu.XXXXXX`
cd $tmpDir
log "Current directory is $tmpDir"
#move and change the permission of the uploaded script
mv $cwd/word_counter .
chmod 755 word_counter

#try to find SQUID location on that site if failed execute srm-copy
if [ x$OSG_SQUID_LOCATION != x ]  &&  [ $OSG_SQUID_LOCATION != UNAVAILABLE ]
then
        log "Found Squid at $OSG_SQUID_LOCATION on `uname`, will try to download file the web"
        export http_proxy=$OSG_SQUID_LOCATION
        wget -d -O/$PWD/$file  http://www.gutenberg.org/ebooks/8813.html.gen  2>&1
        retVal=$?
else
        log "Executing: srm-copy $surl/$file file:///$PWD/$file"
        srm-copy $surl/$file file:///$PWD/$file
        retVal=$?
fi
log "Return value $retVal"
if [ $retVal -eq 0 ]
then
        #move word_counter script
        option=""
        if [ x$logname != x ]
        then
                option="--output=$logfile"
        fi
        log "Executing: $cwd/word_counter $file  $word $option"
        ./word_counter $file  $word --output=$logfile
        retVal=$?
        log "Return value $retVal"
fi
cd $OSG_WN_TMP
rm -rf $tmpDir
log "End test4.sh"
exit $retVal
 

-- TanyaLevshina - 09 Jul 2010

Topic revision: r42 - 10 Oct 2011 - 21:22:17 - JamesWeichel
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..