Data Storage and Management on the OSG

OSG is both a Data Grid and a Computational Grid. That is to say management of data (as well as jobs) is a key part of the architecture and services of the common infrastructure. The OSG Stakeholder requirements on data management (movement, storage, access, tracking, control, monitoring) vary widely:

• Petabyte scale: with a need to store and track up to 10s of petabytes, and stream and store 10s of Terabytes to be accessed locally by or delivered locally from jobs running on OSG sites.
• Terabyte scale: with data needing to be moved to moved 'on the fly' to and from jobs scheduled to run on OSG sites.
• Gigabyte scale: with typically many ancillary files needed by each job in order to operate on the main data store.

The scope of OSG Storage and Data management includes:

• Storage Element implementation, policy, deployment and documentation about storage areas that are accessible to the OSG infrastructure.
  • The storage resources are of clearly-defined types (tape and disk), and flavor (permanent, persistent, transient, and volatile).
  • Interfaces for the writing and reading of data by applications to and from storage over the grid.
  • Policies and agreements for use of Storage Resources on OSG.
• Local Storage implementation, policy, deployment and documentation of storage accessible to the applications when running on an OSG site.
  • Definition of the data and storage areas available in the job's execution environment.
  • Policies and agreements for the use and management of local storage on OSG sites.
• Storage Interface Definitions with respect to interoperability of storage resources across Grids.
• Monitoring and Accounting information and publishing.

Weekly Storage Activity Meetings

Meetings are every Wednesday

• Time: 3:00 pm Central
• Call: 510-665-5437
• Meeting ID: 0655

See also the Minutes from past meetings

Storage and Data Management Elsewhere on the OSG TWiki

Storage Element Documentation is available from the main OSG Documentation Webpage. If you are anEnd User, look under the "End User" section, if you are an Application Developer, look under the "Application Developer" section and if you are a Storage Element Administrator, look under the "Resource and Site Administrators" section. Other SE related documentation is also available on the same webpage.

Questions?

If you have any questions related to Storage Element, please send email to osg-storage@opensciencegrid.org