Accounting And WLCG

Overview

Collaboration with WLCG is typically formalized through negotiating and signing a Memorandum of Understanding (MoU). An MoU puts in place a formal partnership, describing the legal framework and agreements for involvement in WLCG, plus the annexes which are regularly updated according to pledges delivered by the collaborating institutions. The US-WLCG Tier1 and Tier2 facilities have to demonstrate to the WLCG project delivered computing resources on behalf of OSG VO collaborations, in accordance with these signed MoU agreements.

The OSG Gratia accounting service is the basis for the accounting data provided. Accounting data for specific OSG resource groups are summarized on a monthly basis. Wall and CPU consumption are adjusted using HepSpec2006 (HS06) factors for the processor types of the clusters, and then forwarded to an APEL server at the EGI GOC.

The Gratia accounting data being sent to EGI, summarized by month, is:

  • Resource group
  • VO
  • Distinguished Name (subject of the user's proxy)
  • Total number of jobs
  • CPU time (hours)
  • Wall time (hours)
  • Normalized (HS06) CPU time
  • Normalized (HS06) Wall time

This accounting data is used to publish monthly Tier 1/2 reports for those OSG registered resource groups that have been registered with WLCG and have enabled the MyOsg WLCGInformation InteropAccounting flag and AccountingName data.

The OSG VOs which are currently requesting their accounting data be published to EGI are:

  • ATLAS
  • ALICE
  • CMS

Additionally, a VO must specify the OSG resource_groups for which their accounting data should be forwarded. This is necessary because the CPU and Wall times sent are both raw and normalized based on the cluster configuration of each resource_group .

For an OSG resource group (aka LCG site) to be included in WLCG Interoperability reporting the following must occur/exist:

  1. It must be registered with WLCG (see the WLCG Registration section)
  2. It must be registered in OSG (see the OSG Registration section) with the WLCG Interop Accounting data populated for each resource within the resource_group that is to be reported to WLCG.
  3. Verify your accounting data is being reported correctly to the Gratia accounting system.
  4. Verify your data is being reported correctly by GIP.

For a detailed description of the interface, refer to the OSG Accounting APEL/WLCG interface document.

WLCG Registration

An OSG resource group must be registered with the WLCG as a 'site'. When registering in WLCG you need to be defined as part of a Tier2 Federation according to the WLCG MoU. If you do not know which is the relevant federation, contact the US management for the appropriate experiment. You can see the existing site-federation associations in the WLCG REBUS Topology. These should be good sections to get started with:

It is very important to understand that the WLCG Site is equivalent to the OSG resource group and not the resource within the resource group.

Once this registration process is complete, your OSG resource group will be viewable in either the EGI Accounting Portal Tier 1 view or EGI Accounting Portal Tier 2 view.

Additionally, you will be included in the EGEE Monthly MoU Reports issued by the WLCG office. These are generally available on the 8th of the month for the previous month.

IMPORTANT: If you are deprecating (disabling) an OSG resource group (and maybe replacing it with another), you do NOT want to ask the WLCG office to remove the deprecated one. The WLCG REBUS Topology shown for your Federation Accounting Name is not time sensitive. You will lose the historical information for the deprecated OSG resource group (LCG site) if you do.

OSG Registration

A resource group (aka LCG site) must be registered in OSG Information Management System (OIM) with at least one active resource. See the OIM Registration Instructions for details on what is required.

You can have 1 or more resources within that resource group providing any number of services. For each resource you want reported to WLCG, for the resource group, you need to populate the WLCG Information InteropAccounting? data.

The process for enabling Accounting Interoperability when registering (or at later point in time):

  1. Check the box indicating "This is a WLCG resource" (the display will expand to add more information)
  2. Check the box "Enable InteropAccounting?" The purpose of this flag is activate the forwarding of Gratia accounting data for that resource to WLCG accounting for the purpose of MoU accountability and also includes this resource in the "Capacity and Benchmarking Report" sent to osg-wlcg-reports@opensciencegrid.org.
  3. Populate the "WLCG Accounting Name" with the Federation Accounting Name to which your resource group (aka WLCG site) belongs in the WLCG REBUS Topology when you did your WLCG Registration.
  4. In addition, You should also check the box "Enable InteropBDII?" for your resource. This allows GIP/BDII data for this resource to WLCG InteropBDII? (for lcg-bdii.cern.ch) for the purpose of determining a normalization factor for the accounting data sent to APEL/WLCG.

Data will only be forwarded to WLCG for OSG resources that have this InteropAccounting data populated. The accounting data will be summarized by resource group (LCG site) for all resources with the InteropAccounting? flag set to True.

This data will be viewable in the EGI Accounting Portal OSG view. However, it will not be used in any WLCG reporting until the resource group is registered with WLCG (WLCG Registration section).

Verify Gratia reporting.

You will need to verify that the accounting data is being reported correctly to Gratia with regard to:
  • The VO the data is being reported as. The interface only sends data to WLCG for the CMS, ATLAS and ALICE VOs.
  • The Gratia site (MyOsg resource name) your data is being reported as.

To do this, go the the Gratia Accounting Reporting page:

  • In the left hand menu, select "Daily Usage by VO for Site"
  • In the "Site Name" box, select your site (this is the OSG resource you populated with WLCG Interop Accounting data)
  • Then click on the "Display Report below" button.
  • Verify that the results seem reasonable for the VOs being reported.

To trouble-shoot any reporting problems, first review the OSG Validate Gratia documentation.

If you still feel there is a problem with the data being reported, open a GOC ticket:

  • Go to MyOSG
  • Select "GOC Ticket" under "Other Services" on the right hand side.
  • When you are on the GOC Ticket/Submitter page, under "Central Service Issues", select "Gratia".

Verify GIP/BDII reporting

You will need to verify that GIP (Generic Information Provider) is reporting correctly to the OSG BDII service for the OSG Resource in your resource group.

This GIP/BDII data being reported is used to determine a normalization factor for the accounting data sent to APEL/WLCG.

To trouble-shoot any reporting problems, first review the OSG Validate GIP/BDII documentation.

If you still feel there is a problem with the data being reported, open a GOC ticket:

  • Go to MyOSG
  • Select "GOC Ticket" under "Other Services" on the right hand side.
  • When you are on the GOC Ticket/Submitter page, under "Central Service Issues", select "GOC BDII".

Open Issues

Refer to the Accounting Systems Interoperability Questions, Issues, and Proposals twiki

Comments

Topic revision: r38 - 07 Feb 2017 - 19:38:04 - BrianBockelman
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..