Gratia Interfaces - APEL/WLCG

Description

Accounting records collected by Gratia are forwarded to the EGI accounting system, APEL. This is important for US-LHC Tier1 and Tier2 facilities which have to demonstrate to the LHC project delivered computing resources on behalf of ATLAS, CMS and ALICE collaborations, in accordance with signed Memorandum of Understanding agreements. A service outside the OSG software infrastructure (VDT) runs at Fermilab, parses and analyzes accounting records for a resource, scales wall time and cpu usage into HepSpec2006 for the processor type, and then forwards this data to the APEL Server at the EGI GOC

The APEL Accounting Portal at RAL has been storing CPU accounting results for WLCG. On behalf on the ATLAS, CMS and ALICE Tier 1 and Tier 2 that are part of OSG, OSG uploads usage information from Gratia to the APEL accounting system. This OSG data can be viewed in the EGI Accounting Portal - OSG view. APEL is using OSG resource groups registered in OIM (OSG Information Management System) and viewable in MyOSG

Contact Information

For support issues on this interface, the contacts are:

For non-support general type issues on this interface, the contacts are:

Frequency

The update to APEL from Gratia is performed daily, aggregating usage information for the current month on a per resource group and VO basis (and effective 11/1/2009 by User Distinguished Name). Each update is a complete refresh of the current month's usage.

In order to insure completeness of the previous month's usage, a special update is performed for the previous month for the 1st 5 days of the current month. This special update allows for any late reporting to Gratia that may occur. In addition, this been set to coincide with the LCG Accounting Office extract of the data from the EGI accounting portal for the monthly MOU reports. This occurs on the 8th of each month.

These are cron jobs running between 00:00 and 02:00 Central Time daily. The update is performed by deleting the entire months data and then re-inserting (all using sql dml statements) the new summaries.

The APEL system then performs an aggregation of all the accounting data collected from all sites (inclusive of OSG) and updates the EGI Accounting Portal at least once per day. As this process involves a lot of aggregation over a large database the actual time this occurs can vary. The last update time can be viewed at the bottom of the EGI Accounting Portal page.

Accounting Data

Since Gratia handles the collection of detailed accounting data for OSG, this interface sends only monthly summaries for select OSG resource_groups and VOs as shown below:

Data Description
Month  
Year  
Resource Group APEL/EGI Site is equivalent to OSG's OIM resource group
VO VO
User Name User's DN
Number of jobs Total jobs for the period
CPU time Total CPU time (in hours)
Wall Time Total Wall time (in hours)
Normalized CPU CPU Time normalized by the resource group's HEPSPEC06
Normalized Wall Wall Time normalized by the resource group's HEPSPEC06
Earliest End Time End time of the first job in the month
Latest End Time End time of the last job in the month

Details on the complete set of summary data accepted can be found here: APEL Job Summary Records. It should be noted that APEL/EGI is designed to accept VOMS group and role attributes but, as of this time, we have not been asked to provide them.

The Process

As mentioned earlier, the interface reports Gratia accounting data to APEL/EGI for select resource_groups and for only specific VO data for those resource_groups using normalization factors at the resource_group level. Since, at this time, MyOSG does not provide information on this, two manually maintained configuration files are required:

For each resource_group, a Gratia sql query will be constructed using the following filters:

  • Using the lcg-reportableSites file, the process queries MyOSG to determine the set of resources within each resource_group to determine if that resource's accounting data is to be included. It uses the WLCGInformation InteropAccounting flag (True) for this determination.
  • The lcg-reportableSites file provides the normalization factor to be applied.
  • The lcg-reportableVOs file provides the list of VOs whose data is to be reported.

THE Gratia sql query data is then formatted and transmitted to APEL via the SSM protocal.

The sub-sections that follow provide more detail.

Reportable resource groups

This is a manually maintained configuration file (lcg-reportableSites) identifying the resource_groups that are interfaced to APEL/WLCG. This file contains the resource_group and normalization factor used.
#--- CMS Tier 1 ----
USCMS-FNAL-WC1   10264
#--- CMS Tier 2 ----
CIT_CMS_T2          12944
GLOW                9632
    :
#--- ATLAS Tier 1 ----
BNL_ATLAS          12372
#--- ATLAS Tier 2 ----
AGLT2               8500
HU_ATLAS_Tier2      8872
    :
#--- ALICE Tier 2 ---- 
NERSC-PDSF          13920
LC-glcc             15680

Normalization Factor

As you've seen in the APEL accounting data section, the data sent to the APEL/WLCG data base contains both a raw and normalized value. The WLCG requires a normalization factor to be applied. It is felt that this is the only fair means of comparing data across a wide variety of grid clusters. Prior to January 2011, the normalization factor was based on SpecInt2000 benchmark values for a processor and the monthly MOU reports reflected that. Starting in January 2011, the MOU reports and Tier 1 spreadsheets reflect HepSpec2006 benchmarks.

The normalization factor is basically a simple weighted average of a resource_group's sub-cluster configuration. For example:

Processor Cores Benchmark (Cores * Benchmark)
Intel(R) Pentium(R) 4 CPU 2.60GHz 120 978 117,360
Dual Core AMD Opteron(tm) Processor 270 102 1,452 148,104
Summary: (2 Subclusters) - Totals 222 265,464
Normalization Factor (265464 / 222) 1,196

As newer processors were put into production, a problem started to emerge in that SpecInt benchmarks were unavailable and only HepSpec benchmarks were. On the other side, there were no HepSpec values available for the older processors. In addition, given the current interface, there was no way of distinguishing between a SpecInt and HepSpec derived normalization factors and there was really no way to compare the two values, a decision was made by the WLCG to apply a factor of 4 as an arbitrary equalization factor for the two benchmarks.

However, the interface, as described in the APEL accounting data section, only provides for normalized data using a HEPSpec2006 benchmark:

  • if the processor has a SpecInt2000 value, we mdiultiply by 4 and use that.
  • if the processor has a HEPSpec2006 value, we use that.

Then, when the data is sent to APEL and subsequently to the EGI portal, the HEPSpec2006 normalized values are divided by 4 to reflect the SpecInt200 normalized values. This can be seen under any of the views in the EGI Accounting portal like EGI Accounting Portal - OSG view when you select the Data to graph pull down at the top. At some point, SpecInt2000 should be totally deprecated, however, that will be transparent to Gratia and this interface.

Reportable VO data

A manually maintained configuration file (lcg-reportableVOs) identifying the VOs whose accounting data is interfaced. Only accounting data for the CMS, ATLAS and ALICE VOs is sent to APEL/WLCG for each of the resource_groups at this time.
cms
uscms
atlas
usatlas
alice

Gratia query

This is the Gratia sql query used for each resource_group defined in the lcg-reportableSites file. The criteria, defined in the previous sections, is highlighted in red.
SELECT 'RESOURCE_GROUP'   as Site,
   VOName   as "Group",
   min(UNIX_TIMESTAMP(EndTime))  as EarliestEndTime,
   max(UNIX_TIMESTAMP(EndTime)) as LatestEndTime,
   date_format(min(EndTime),"%m") as Month,
   date_format(min(EndTime),"%Y")  as Year,
   IF(DistinguishedName NOT IN (\"\", \"Unknown\"),
      IF(INSTR(DistinguishedName,\":/\")>0,
      LEFT(DistinguishedName,INSTR(DistinguishedName,\":/\")-1), DistinguishedName),CommonName) as GlobalUserName,
   Round(Sum(WallDuration)/3600)                        as WallDuration,
   Round(Sum(CpuUserDuration+CpuSystemDuration)/3600)   as CpuDuration,
   Round((Sum(WallDuration)/3600) * NORMALIZATION_FACTOR ) as NormalisedWallDuration,
   Round((Sum(CpuUserDuration+CpuSystemDuration)/3600) * NORMALIZATION_FACTOR) as NormalisedCpuDuration,
   Sum(NJobs) as NumberOfJobs
from
     Site,
     Probe,
     VOProbeSummary Main
where
      Site.SiteName in (LIST_OF_MYOSG_RESOURCES_WITH_INTEROPACCOUNTING_FLAG_TRUE)
  and Site.siteid = Probe.siteid
  and Probe.ProbeName  = Main.ProbeName
  and Main.VOName in ( LIST_OF_REPORTABLE_VOS )
  and %(period)s
  and Main.ResourceType = "Batch"
group by Site,
         VOName,
         Month,
         Year,
         GlobalUserName

The reason for the complexity in extracting DistinguishedName in Gratia is two-fold:

  1. the check for empty or 'Unknown' is because the DistinguishedName was not available in Gratia until 11/1/2009
  2. the even more complex exclusion of everything after the first ":/' was to compensate for a new feature in Condor 7.3.2 and 7.4 that captured the VOMS proxies extended attributes and appended them to the x509userproxysubject attribute which is the source of the data.

The filter on ResourceType was due to a change in the condor probes (v1.00.3) to distinguish between the actual grid jobs (Batch) and the grid monitoring job (GridMonitor) when jobs are submitted using condor_submit. Any 'local' job used to submit a job on the CE node will be filtered, but should they at some point be passed to Gratia, these will be identified as 'Local'.

Validation/Verifications

There are several integrity checks that the LCG.py process makes to insure all data is being reported correctly.

Sites/resources are reporting.

As a part of the interface process, a check is made at the site/resource level (not individual probes) to determine if data is being reported to Gratia every day. If a site/resource has missing data for any day, checks are made to see if this is due to planned maintenance or if a site/resource has gone inactive based on OIM/MyOSG data. If after these checks are made and a site/resource still has missing data for more than 2 days in the month, an email "WARNING" notification is issued to initiate an OSG GOC ticket. Example:
WARNING/ADVISORY: Resource: UCSDT2-B in Resource Group: UCSDT2  missing data for more than 2 days: ['2014-03-19', '2014-03-20', '2014-03-21',]
Note: this is a manual ticket initiation.

If, for whatever reason MyOSG is unavailable, an additional warning message will be generated advising of this. This is not a critical problem unless it persists for several. When this occurs, any messages regarding sites not reporting should not be acted upon.

  • WARNING: Unable to determine planned shutdowns

MyOSG and EGI REBUS topology agree

Since, at the current time, only those OSG resource_groups registered in the EGI REBUS topology will be reported for MOU purposes, a validation is made to insure MyOSG and the EGI REBUS topology are in sync. The terminology between the two is slightly different:

Rebus MyOsg
Federation Accounting Name WLCGInformation Accounting Name
Site resource group
A query is made to EGI REBUS topology and against MyOSG comparing the MyOSG WLCGInformation AccountingName with the Rebus AccountingName to insure that monthly MOU reporting will be accurately for the resource_group's data the interfaced sends to APEL. This should be investigated and a goc ticket should be generated to take correct the descrepancy. There are too many combination of conditions to highlight here. They are self explanatory as to the problem.

Note: this is a manual ticket initiation/process

The following warning can occur and are not immediately critical unless they persist:

  • WARN: The WLCG REBUS topology was not accessible today. We are using a previous days data for validations

Other warning/error messages

On the first day of each month, a warning message will be generated advising that the following files should be added to svn:
  • the current months lcg-reportableSites file
  • These files should be checked to see if any updates should be made to SVN/CVS in order to retain their history.

Maintaining the Normalization Factors

Updating of the normalization factor for a resource_group is a manual process.

The ALICE and CMS sites provide their own normalization factor.
ATLAS maintains their own calculations manually and can be viewed here:

A goc ticket should be generated to initiate a change.

When a change is made, svn should be updated for both the main /etc/gratia-apel/lcg-reportableSites file and the appropriate /etc/gratia-apel/lcg-reportableSites.history file.

Maintaining the Reportable Sites and VOs

This is also a manual process. As seen in the section on warning messages, there is a process which checks MyOSG at the resource level for the InteropAccounting flag and warns if a site says it is reporting but has not yet been added to the lcg-reportableSites file.

A goc ticket should be generated to investigate.

Would like to have enhancements

Move the manually maintained configuration files to MyOsg

Eventually the lcg-reportableSites file should be replaced with data in MyOSG.

The areas that would have to be addressed are:

  1. Identification of the resource_groups that are to be reported/interfaced.
  2. Maintenance of the normalization factor at the resource_group level
  3. Identification of the VOs to be reported/interfaced

Open issues / To-Do's

Synchronization of OIM/MyOSG with Gratia

The intent of this initiative is to align OIM/MyOSG with Gratia and, additionally, extend that to the APEL/WLCG interface. The motivation is to eliminate the manual intervention required when new resources (Gratia sites) are added or removed from service. Currently, there does not exist any true relationship between resources in OIM/MyOsg and Gratia. The OIM/MyOSG resource name should correspond to a Gratia site name. In the following paragraphs, when the term 'resource' is used, it is considered synonymous with Gratia 'site name'.

In OIM/MyOSG, the ability to define resources that are involved in the APEL/WLCG interface currently exists This data is at the resource level and contained in the WLCG Information section:

  1. Interop Accounting flag - True if the resource is interfaced to APEL/WLCG for accounting purposes
  2. Accounting Name - if the Interop Accounting flag is True, this contains the Tier 1 or 2 WLCG federation name (e.g., T2-US-Caltech)

From an APEL/WLCG perspective, we need to handle the following use cases:

  1. Addition of a new resource for a Tier1/2 Federation
  2. Removal of a resource from a Tier1/2 Federation (aka, disabling a resource in OIM/MyOSG terms). This requires preserving the historical data for that resource.

Validation/verification of monthly MOU reports

When the Tier 2 monthly reports are issued by the LCG accounting office (e.g, http://lcg.web.cern.ch/LCG/accounting/Tier-2/2010/january-10/Tier2_Accounting_Report_January2010.pdf), a validation of the OSG federation resource is being performed manually. It would be "nice" to be able to able perform some form of automated validation and eliminate this manual effort. I am not sure if this is totally viable.

Normalization factor issues

  1. As far as Gratia/Accounting is concerned, we would like to get to a point where each job usage record contain the type of cpu that the job ran on. This would give Gratia the most flexibility to normalize the result (normalization is required by WLCG and is the only way to fairly compare the result).

    However, at this point the batch system does not gives us enough information (Condor 6.9 might be able to make the information available). So for now we will rely for normalization on a rougher estimate. Currently the estimate comes from the average on one existing farm for which I know the actual hardware. When the GLUE schema is properly filled, we will use the average for the Site the job ran on.

  2. The normalization factor is currently reflecting the latest cluster configuration. As these clusters are upgraded over time, these factors will change. In order to be able to re-create the normalized view of past periods, these configurations/factors will need to be keyed by time period somehow.

History

This section just documents when major changes to the interface took place:
Date Description
September 2008 Added methods to create xml/hmtl/dat files to provide visibility to the data sent and displayed at Gratia-APEL WLCG Interface and OSG WLCG Reporting
November 2008 Added an additional filter on the selection criterea. Only selecting ResourceType='Batch'. Changes to the condor probes (v1.00.3) will result in distinguishing between the actual grid jobs (Batch) and the grid monitoring job (GridMonitor) when jobs are submitted using condor_submit. Any 'local' job used to submit a job on the CE node will be filtered, but should they at some point be passed to Gratia, these will be identified as Local.
March 2009 Started saving the configuration file for reportable sites and normalization factors for each month. This provided the capability to re-send prior months data in the event a site had problems and still retain that period's configuration.
May 2009 Added the User's Distinguished Name (DN) as part of the summary criterea and data sent.
June 2009 Started checking for Gratia sites that had stopped reporting data to Gratia. This provided a pro-active means of identifying problems rather than waiting for the monthly MOU reports to be issued.
July 2009 Added an extract of OIM downtimes to use when determining if a Gratia site failed to report
December 2009 Added use of a new class (InactiveResources) to query MyOsg? for Resource that have been marked inactive. This is used in conjunction with the Downtimes class when checking why a site/resourcehas not reported
January 2010 APEL/EGI started reporting MOU reports using HEPSpec2006 normalization factors in place of the SpecInt2000 values we are reporting. We still use SpecInt2000. They apply a factor of 4x as means of adjusting until all sites report using HEPSpec2006. When we have HEPSpec2006 available we divide by 4 to maintain consistency.
December 2010 The ALICE VO was added to the list of reportable VOs and sites.
August 2011 This represents a major change to the interface. All OSG reporting to WLCG has been in reference to MyOsg resource group. However, the MyOsg InteropAccounting flag for WLCG Information is at the resource, not resource group, level. The changes made in this revision will now:
1. treat the lcg-reportableSites config file entries as resource groups. This also means the Normalization Factors should be calculated at the resource group level.
2. determine which resources within that resource group should have their gratia data reported to APEL/EGI.
3. summarize (using sql) the accounting data for the month for all interfaced resources for the resource group. A new class, InteropAccounting(.py), provides the access to MyOsg.
June 2012 Changed the APEL interface from a direct update of the APEL database to the use of a messaging system (SSM)

More detail information related to code, reportable sites and normalization factors can be found in the Gratia Interfaces - APEL/WLCG - History twiki

More information

Sending Data to EGI Based on VO Only

APEL SSM/ActiveMQ Interface

Developers Corner

EGI

Major updates

-- JohnWeigand - 19 Nov 2009: Updated for implementation of User Distinguished Name on 11/1/2009
-- JohnWeigand - 08 Feb 2010: Split the developers corner (doc) out as a separate twiki (ApelWlcgDevelopersCorner)
Topic attachments
I Attachment Action Size Date Who Comment
xlsxls Normalization-factors.xls manage 38.0 K 21 Mar 2008 - 20:30 UnknownUser Normalization factor calculations
xlsxls OIM-registered-resource-3-5-2009.xls manage 48.2 K 19 Mar 2009 - 16:11 JohnWeigand OIM Registered Resources 3/5/2009 (.xls format)
elseEXT lcg-reportableSites manage 6.8 K 21 Mar 2008 - 20:29 UnknownUser APEL/WLCG site table
Topic revision: r66 - 27 Mar 2014 - 15:21:59 - JohnWeigand
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..