Storage Meeting Minutes 2009 Sep 30
- Andrew Baranovski
- Brian Bockelman
- Ted Hesselroth
- Neha Sharma
- Alex Sim
- Tanya Levshina
Moved SCEC support forward by encouraging UFL to fix their GIP configuration.
Though the GIP team will not offer support for their non-hep compute element due to extensive customization,
YuJun Wu is working on this. So, kudos.
Made and tested several gPlazma patches for dCache 1.9.5 "Golden" release. Instead of returning only the
authorizations, gPlazma now returns a map of all roles found and their authorizations, including "null" if
the role is not authorized. Solved a problem of unexpected results when multiple roles mapped to the
Follow up on tickets.
- srm-copy usage for a large file/directory transfer
- srm-ping and srm-copy usage in RSV probe
* bestman 184.108.40.206 release
- packaging, testing
* next major release as bestman2 will be towards the end of this year.
bestman2 uses tomcat for the http container and the latest
current bestman will be maintained for a while as bestman1
until all sites move to bestman2, but all new features will only be
added to the bestman2.
* bestman new features in 220.127.116.11
1. VOMS validation is supported
conf entry: validateVomsProxy=true (false by default)
With or without gums support in the server, it validates the voms proxy.
2. user space keys are supported
userSpaceKeywords is to define keys to shorten the path
bestman will disable space management when this is defined
and SFN=/tic/fpath1 in SURL will be translated to SFN=/user/tic/fpath1
difference from staticTokens: space descriptions are not supported.
3. checksum callout is supported
this is effective only if showChecksumWhenListingFile=true
conf entry: hexChecksumCommand
e.g. hecChecksumCommand=/usr/local/bin/mychecksum -adler32
When a file needs checksum, the file path is attached to the call-out
and the returns can be seperated by spaces, and
the first value is assumed to be the checksum value output.
4. No directory browsing is supported as a server side enforcement.
conf entry: NoDirBrowsing=true
It will handle srmLs on all directory level browsing with level 0
returning the directory information only.
* bestman changes in 18.104.22.168
5. (full mode) fixed wrong space token generation after restarting the
6. (full mode) fixed wrong status code return in srmCopy
7. (full mode) fixed truncated directory browsing for a long path name
* setup changes and new options in 22.214.171.124
1. all command line scripts support dash environment on Ubuntu
2. added --with-container-threads= for web service container threads
defaults to 64 per 512MB max java heap size
3. added --enable-voms-validation
4. added --with-user-space-key= for user space keys
5. added --enable-voms-validation
6. added --with-checksum-callout= to specify path for checksum
7. added configure.gateway and configure.fullmode
conf/bestman.rc will be customized and simplied for each mode.
* srm clients lbnl - new features in 126.96.36.199
1. added clear status tags (request or file) for returned status information
2. added "-connectiontimeout " in srm-copy for soap or gsi
timeout as well as file transfer timeout.
Tier-3 liaison group
OSG VO forum
OSG Documentation meeting
Xrootd/Xrootd-GridFTP/Xrootdfs release/support policy
Status of SCEC jobs with Michael, Brian
VOMS Validation in new BeStMan release
Comments for standalone gridftp document
dcache support: upgrade discussion with David Lesny
comments on request to include details for configuring service certificate for BeStMan in documentation
discussed workaround for xrootd problem i've encountered
Applied workaround suggested by xrootd developers, was able to "restore" file meta data , has recommended
to release xrootd/xrootdfs for vdt
Tested new release of BeStMan, BeStMan client and srm tester.
Started to work on thridparty transfer tests
Reviewed Andrew's proposal, provided feedback
Andrew's troubleshooting/log collection design doc - any feedback?
Brian has questioned the viability of our decision to look into log collection/troubleshooting. He thought that the similar attempts have failed before (e.g. he has mentioned about apparent reluctance of sys admins to install syslog-ng for log collection). Tanya reminded that we have the following item in our WBS for FY10:
"Investigate tools for storage troubleshooting"
Brian agreed that this activity seems to be relevant to WBS and asked Andrew to add more implementation details into the document. Tanya asked to add real use cases to the document, e.g example how logs and relevant filtering rules could help troubleshooting. Andrew said that he would need help with that.
Storage discovery tools - how far we from VDT release?
Ted started working on improving caching. He will contact STG to report the progress and discuss packaging. Tanya suggested to provide configuration script that would allow to configure discovery tools to access OSG or ITB bdii.
Progress with SCEC (Ted)
Mats loaded tar files to Caltech, Nebraska and Firefly. Now, he will need to untar them and run jobs. UFlorida is still working on GIP configuration.
New Bestman release and plans for the next major release (Alex)
is released and ready for testing (see status report). BeStMan2?
work has started.
Hadoop, space tokens & space utilization plugin
Plugin is needed to report space usage for static tokens (lcg-cp failed if this information is not available). Brian said that there is a way to set quota on hadoop fs and use hadoop command to report space utilization.
Andrew will try to set quotas on our local test stand and write plugin for BeStMan
RSV Storage Probes
Arvind has requested that OSG Storage takes responsibilities over RSV Storage Probes. We will need to talk to Alain and understand if this is a reasonable request for us before making any decision.
7573 - Hadoop - Root I/O Plugin for hadoop
Follow up in progress:
7571 - Bestman - GSIFTP timeout with srm-copy
7572 - Hadoop - Tuning Tips
7380 -dCache - gPlazma drops roles when multiple roles map to same username
7447 - dCache - Low throughput observed after upgrading to dcache 1.9.4-3
Follow up in progress:
7505/7574 - dCache - Site losing data due to files being marked as non-sticky
7504 - dCache - Configuring space tokens at user account level instead of VO/Role
7448 - dCache - Poor PNFS/Chimera performance on upgrade from 1.9.2-5 to 1.9.4.-x
6967/7010 - dCache - File replicas not being removed
7326 - dCache - misleading error message when using dccp to transfer data
6971 - dCache - Problem with pnfs register command
6908 - dCache - log4j errors when restarting dcache 1.9.2-5
- 01 Oct 2009