Gratia Gratia Accounting Reports for OSG and Fermilab Short Description Introduction Contact information and mailing lists Meetings Metrics Goals Monitoring vs Accounting Installation Probes Collector and Reporting VDT Installation Instructions VDT Packaging and Installation Instructions Local FNAL-specific installation guide Requirements Reporting Requirements Operational Requirements Gratia Login/Access Requirements Implementation (developers' corner) Specifications Entity Relationship Diagrams (ERD) Development and Test Environment Production database conversion procedure from MyISAM to INNODB Other internal documents Notes on the OSG Requirement documents WBS Production Releases Building a new release for distribution Gratia Upgrade Procedure - under construction Gratia Release v0.28 - 10/27/2007 Gratia Release v0.30 - 11/20/2007 Gratia Release v0.32 - 2/1/2008 Gratia Release v0.32.1 InnoDB - 2/14/2008 -4/2/2008 Gratia Release v0.32.1c - 3/31/2008 - 4/1/2008 Gratia Release v0.34 / v0.34.9 - 5/10/2008 - 6/29/2008 Gratia Release v0.36 - 07/15/2008 Gratia Release v0.38 - 09/16/2008 Gratia Release v1.00 - 10/20/2008 Gratia Service Operation Interfaces APEL-WLCG Code Documentations Readiness Plan for OSG 0.6 References and Resources

Gratia Accounting Reports for OSG and Fermilab

Reports can be viewed at several places depending on the type of data being collected:

• Production OSG sites
• OSG daily summaries (legacy)
• ITB and testing
• Fermilab's PS-accounting information (Fermi-specific)
• Fermilab's LQCD batch system (non-OSG)

These reports are implemented using the open source Business Intelligence and Reporting Tools - BIRT.

For a more detailed description on the type and use of the reports please refer to GratiaReportGuide

Short Description

The Gratia Project (contributed as an OSG Activity) designs and deploys robust, scalable, trustable and dependable grid accounting, publishes an interface to the services and provides a reference implementation.

OSG Job

Reports issues and new features request via sourceforge

Introduction

The Grid Accounting Activity is designing and deploying robust, scalable, trustable and dependable grid accounting services, publishing an interface to the services and providing a reference implementation.

Contact information and mailing lists

• The project lead is Philippe Canal.

• Reports issues and new features request via Sourceforge

• Problem reports with existing or new installations of probes or collectors should go to the Gratia Operations mailing list:
  • Send mail to gratia-operation@opensciencegrid.org.
  • Browse the Gratia Operations list archive.

• General discussion takes place on the OSG Accounting mailing list:
  • Send mail to osg-accounting@opensciencegrid.org.
  • Subscribe to osg-accounting@opensciencegrid.org.
  • Browse the OSG Accounting list archive.

• Daily email reports from the DB are sent to the OSG Accounting Info mailing list:
  • Subscribe to osg-accounting-info@opensciencegrid.org.
  • Browse the OSG Accounting Info list archive.

• Finally, other matters that you may wish to raise particularly with the developers should be sent to our internal mailing list:
  • Send mail to grid-accounting@fnal.gov.

Meetings

The Gratia team meets weekly on Wednesday at 1:30pm CST. To Join call 510-665-5437 and use the meeting ID: 2966.

See Meeting Minutes

Metrics

See the Metrics page.

Goals

The main goal of the Grid Accounting Joint Project, which is additionally being contributed as an OSG Activity, is to provide the stakeholders with a reliable and accurate set of views of the Grid resources usage.

The Grid Accounting Project:

• has designed the schema for the accounting attributes,
• is ensuring the necessary collectors and sensors are in place in the resource providers,
• has defined and is deploying repository and access tools for the reporting and analysis of the grid wide accounting information.

The Accounting system will properly determine a confidence level in the existing accounting information and adequately address and present erroneous or missing accounting data.

The Accounting system will adequately protect the privacy of the users and organizations involved.

The initial main goal for the accounting system will be to track VirtualOrganizations?/VOInfo members' resource usage and to present that information in a consistent Grid-wide view, focusing in particular on CPU and Disk Storage utilization.

Full Project Definition

See the current list of priorites and the current list of enhancement requests

Monitoring vs Accounting

For monitoring,the goal is to have a view of the current status of the system with minimal delay. Small numerical errors, approximation and statistical estimation in the data are not fatal for monitoring. For example if a CPU is utilized at 82% or 85% does not matter to the monitoring system. For monitoring, missing informatio is not really an issue. For example, the measurement of the cpu load is not really affected if we are missing a few samples.

For accounting, the goal is to have an accurate view of the resource and service usage in the system. This includes linking them to the individual user request. The accounting should not loss any accounting data or record. The main focus is on the historical view and a delay in the information is acceptable. However the information need to be as accurate as possible, and include an estimate of its accuracy. The accounting system need to be able to support the development of an economic mode.

The type of information and the means of collecting this information are similar between monitoring and accounting but the choices of trade-offs are clearly different; for example timeliness vs. accuracy and completeness.

Installation

Probes

Installation of probes may vary by probe, but in general should be done via VDT if available. Please see the probe installation topic for full instructions and further notes.

Collector and Reporting

VDT Installation Instructions

Intended for remote/local collectors that are installed from the VDT pacman cache.

• VDT Installation Instruction

VDT Packaging and Installation Instructions

Intended to be used only by the VDT team for packaging the Gratia services

• VDT-team Gratia service installation guide

Local FNAL-specific installation guide

Intended solely for local installation here at FNAL with multiple collectors.

• Local FNAL-specific installation guide

Requirements

We presented a summary of the Accounting System requirement in July 2005 in Milwaukee. OSG Accounting Requirements Presentation We also presented a general overview of the design (See the same link).

Reporting Requirements

Summary of the ReportRequirements that have been outlined in the above documents.

Operational Requirements

See the discussions and documents in OperationalRequirements.

Gratia Login/Access Requirements

Access to Gratia administration and reporting features and view are defined in the Gratia Login/Access Specification document.

Implementation (developers' corner)

Specifications

PSACCT Probe Design Specification

Gratia Collector Web Service Development, Deployment and Integration

CPU Usage Accounting

Current Gratia Reporting Design Specification

Current Gratia Report Development, Deployment and Integration

Proposed Gratia Reporting Infrastructure Upgrade Implementation Plan

Entity Relationship Diagrams (ERD)

The ERD (Entity Relationship Diagram) for the Gratia database can be see in the formats listed below. It should be noted that it is not a "complete" picture of all the relationships. Many of the foreign key relationships were deliberately left off in order to simplify the diagram. The intent is to show the main parent/child relationships only.

• Gratia v1.00 database ERD (png)
• Gratia v0.38 database ERD (png)
• Gratia v0.36 database ERD (png)
• Gratia v0.34 database ERD (png)
• Gratia v0.30 database ERD (png)

The software used to maintain the ERD diagram is called DeZign for Databases Professional Version 4.2.0 from Datanamic Solutions BV (http://www.datanamic.com). The source file for the various versions is in the Gratia CVS repository in ./gratia/docs/erd :

• Gratia database ERD v100 source: gratia-v100.dez
• Gratia database ERD v038 source: gratia-v038.dez
• Gratia database ERD v036 source: gratia-v036.dez
• Gratia database ERD v034 source: gratia-v034.dez
• Gratia database ERD v030 source: gratia-v030.dez
• Gratia database ERD v028 source: gratia-v028.dez

Development and Test Environment

A description of the development and test environment can be found in the Gratia Development and Test Environment page.

Production database conversion procedure from MyISAM to INNODB

The latest version of the database conversion procedure may be found here; the original draft may be seen here.

Other internal documents

The case for InnoDB ( in progress).

Notes on the OSG Requirement documents

WBS

A draft of the WBS is available.

Production Releases

A Gratia Release twiki template has been created that can be used as the basis for documenting production releases. You can cut/paste from it into a new twiki release document and add specifics for that release. Please do not change the template unless some general release changes are made that are not unique to a release.

Building a new release for distribution

Gratia Upgrade Procedure - under construction

Gratia Release v0.28 - 10/27/2007

Gratia Release v0.30 - 11/20/2007

Gratia Release v0.32 - 2/1/2008

Gratia Release v0.32.1 InnoDB - 2/14/2008 -4/2/2008

Gratia Release v0.32.1c - 3/31/2008 - 4/1/2008

Gratia Release v0.34 / v0.34.9 - 5/10/2008 - 6/29/2008

Gratia Release v0.36 - 07/15/2008

Gratia Release v0.38 - 09/16/2008

Gratia Release v1.00 - 10/20/2008

Gratia Release v1.02 - Early 2009

Gratia Release v1.04 - not scheduled

Gratia Service Operation

All information required to install and operate the official OSG Gratia services may be found through the Gratia Service Operation Guide page.

Interfaces

APEL-WLCG

APEL-WLCG Interface

Revisions APEL-WLCG Interface (a work-in-progress 5/1/2008)

Code Documentations

• How to write a Gratia probe.
• Issue related to detecting a probe black-out (missing information).

Readiness Plan for OSG 0.6

See AccountingReadinessPlan

References and Resources

OSG Accounting Requirements

Global Grid Forum Usage Record Working Group

XML Usage Record definition from GGF

GENERAL

• Barry Varley, Usage Accounting in Distributed Stystems, for presentation at IEE Colloquium on Network Management, 4th October 1991
• B. Aboba, J. Arkko, D. Harrington, Introduction to Accounting Management, RFC 2975, October 2000

GLUE SCHEMA

• http://glueschema.forge.cnaf.infn.it/

AUSTRALIAN GRID

• Rajkumar Buyya and David Abramson and Jonathan Giddy. A Case for Economy Grid Architecture for Service Oriented Grid Computing. 10th IEEE International Heterogeneous Computing Workshop 2001
• A. Barmouta and R. Buyya, GridBank: A Grid Accounting Services Architecture (GASA) for Distributed Systems Sharing and Integration, 26th Australian Computer Science Conference (ACSC2003) Adelaide, Australia Feb 2003.

NORDUGRID & SWEGRID

• Peter Garfjall, Accounting in Grid Environments, an architectural proposal and a prototype implementation, Master Thesis, 27 May 2004, Umea University, Sweden.
• T. Sandholm, P. Gardfjell, E. Elmroth, L. Johnsson, and O. Mulmo. An OGSA-Based Accounting System for Allocation Enforcement across HPC Centers. Proceedings of the 2nd International Conference on Service Oriented Computing. New York, USA, 15-19 November, 2004. http://www.icsoc.org
• E. Elmroth, P. Gardfjell, O. Mulmo, and T.Sandholm. An OGSA-Based Bank Service for Grid Accounting Systems. In J. Wasniewksi et. al. (eds). Applied Parallel Computing. State-of-the-art in Scientific Computing. Springer Verlag, Lecture Notes in Computer Science, 2004. http://www.imm.dtu.dk/~jw/para04/
• http://www.sgas.se

EGEE and gLITE

• APEL
  • http://goc.grid-support.ac.uk/gridsite/accounting/
• DGAS
  • http://server11.infn.it/workload-grid/docs/DataGrid-01-TED-0126-1_0.pdf
  • http://www.to.infn.it/grid/accounting

TERAGRID

• http://www.teragrid.org/userinfo/access/usage.php

FACILITY GROUP REVIEW REPORT

• The OSG Facility Group is reviewing the daily reports produced by Gratia. GratiaReportReviewPage

Reviewed by Marcia Teckenbrock - 03 Jan 2007

• Add definition of Gratia
• Meetings schedules and minutes

-- ChrisGreen - 19 Sep 2007