Gratia Interfaces - APEL/WLCG
Accounting records collected by Gratia are forwarded to the EGI accounting system, APEL. This is important for US-LHC Tier1 and Tier2 facilities which have to demonstrate to the LHC project delivered computing resources on behalf of ATLAS, CMS and ALICE collaborations, in accordance with signed Memorandum of Understanding agreements. A service outside the OSG software infrastructure (VDT) runs at Fermilab, parses and analyzes accounting records for a resource, scales wall time and cpu usage into HepSpec2006 for the processor type, and then forwards this data to the APEL Server
at the EGI GOC
The APEL Accounting Portal at RAL has been storing CPU accounting results for WLCG. On behalf on the ATLAS, CMS and ALICE Tier 1 and Tier 2 that are part of OSG, OSG uploads usage information from Gratia to the APEL accounting system. This OSG data can be viewed in the
EGI Accounting Portal - OSG view
. APEL is using OSG resource groups
registered in OIM (OSG Information Management System)
and viewable in MyOSG
For support issues on this interface, the contacts are:
For non-support general type issues on this interface, the contacts are:
The update to APEL from Gratia is performed daily, aggregating usage information for the current month on a per resource group
and VO basis (and effective 11/1/2009 by User Distinguished Name). Each update is a complete refresh of the current month's usage.
In order to insure completeness of the previous month's usage, a special update is performed for the previous month for the 1st 5 days of the current month. This special update allows for any late reporting to Gratia that may occur. In addition, this been set to coincide with the LCG Accounting Office extract of the data from the EGI accounting portal for the monthly MOU reports. This occurs on the 8th of each month.
These are cron jobs running between 00:00 and 02:00 Central Time daily. The update is performed by deleting the entire months data and then re-inserting (all using sql dml statements) the new summaries.
The APEL system then performs an aggregation of all the accounting data collected from all sites (inclusive of OSG) and updates the EGI Accounting Portal
at least once per day. As this process involves a lot of aggregation over a large database the actual time this occurs can vary. The last update time can be viewed at the bottom of the EGI Accounting Portal
Since Gratia handles the collection of detailed accounting data for OSG, this interface sends only monthly summaries for select OSG resource_groups
as shown below:
| Resource Group
|| APEL/EGI Site is equivalent to OSG's OIM resource group
| User Name
|| User's DN
| Number of jobs
|| Total jobs for the period
| CPU time
|| Total CPU time (in hours)
| Wall Time
|| Total Wall time (in hours)
| Normalized CPU
|| CPU Time normalized by the resource group's HEPSPEC06
| Normalized Wall
|| Wall Time normalized by the resource group's HEPSPEC06
| Earliest End Time
|| End time of the first job in the month
| Latest End Time
|| End time of the last job in the month
Details on the complete set of summary data accepted can be found here: APEL Job Summary Records
It should be noted that APEL/EGI is designed to accept VOMS group and role attributes but, as of this time, we have not been asked to provide them.
As mentioned earlier, the interface reports Gratia accounting data to APEL/EGI for select resource_groups
and for only specific VO data for those resource_groups
using normalization factors at the resource_group
level. Since, at this time, MyOSG
does not provide information on this, two manually maintained configuration files are required:
For each resource_group
, a Gratia sql query
will be constructed using the following filters:
- Using the lcg-reportableSites file, the process queries MyOSG to determine the set of resources within each resource_group to determine if that resource's accounting data is to be included. It uses the WLCGInformation InteropAccounting flag (True) for this determination.
- The lcg-reportableSites file provides the normalization factor to be applied.
- The lcg-reportableVOs file provides the list of VOs whose data is to be reported.
THE Gratia sql query
data is then formatted and transmitted to APEL via the SSM protocal.
The sub-sections that follow provide more detail.
Reportable resource groups
This is a manually maintained configuration file (lcg-reportableSites
) identifying the resource_groups
that are interfaced to APEL/WLCG. This file contains the resource_group
and normalization factor used.
#--- CMS Tier 1 ----
#--- CMS Tier 2 ----
#--- ATLAS Tier 1 ----
#--- ATLAS Tier 2 ----
#--- ALICE Tier 2 ----
As you've seen in the APEL accounting data
section, the data sent to the APEL/WLCG data base contains both a raw and normalized value. The WLCG requires a normalization factor to be applied. It is felt that this is the only fair means of comparing data across a wide variety of grid clusters. Prior to January 2011, the normalization factor was based on SpecInt2000 benchmark values for a processor and the monthly MOU reports reflected that. Starting in January 2011, the MOU reports and Tier 1 spreadsheets reflect HepSpec2006 benchmarks.
The normalization factor is basically a simple weighted average of a resource_group's
sub-cluster configuration. For example:
|| (Cores * Benchmark)
| Intel(R) Pentium(R) 4 CPU 2.60GHz
| Dual Core AMD Opteron(tm) Processor 270
| Summary: (2 Subclusters) - Totals
| Normalization Factor (265464 / 222)
As newer processors were put into production, a problem started to emerge in that SpecInt benchmarks were unavailable and only HepSpec benchmarks were. On the other side, there were no HepSpec values available for the older processors. In addition, given the current interface, there was no way of distinguishing between a SpecInt and HepSpec derived normalization factors and there was really no way to compare the two values, a decision was made by the WLCG to apply a factor of 4 as an arbitrary equalization factor for the two benchmarks.
However, the interface, as described in the APEL accounting data
section, only provides for normalized data using a HEPSpec2006 benchmark:
- if the processor has a SpecInt2000 value, we mdiultiply by 4 and use that.
- if the processor has a HEPSpec2006 value, we use that.
Then, when the data is sent to APEL and subsequently to the EGI portal, the HEPSpec2006 normalized values are divided by 4 to reflect the SpecInt200 normalized values. This can be seen under any of the views in the EGI Accounting portal like EGI Accounting Portal - OSG view
when you select the Data to graph
pull down at the top. At some point, SpecInt2000 should be totally deprecated, however, that will be transparent to Gratia and this interface.
Reportable VO data
A manually maintained configuration file (lcg-reportableVOs
) identifying the VOs whose accounting data is interfaced. Only accounting data for the CMS, ATLAS and ALICE VOs is sent to APEL/WLCG for each of the resource_groups
at this time.
This is the Gratia sql query used for each resource_group
defined in the lcg-reportableSites
file. The criteria, defined in the previous sections, is highlighted in red
SELECT 'RESOURCE_GROUP' as Site,
VOName as "Group",
min(UNIX_TIMESTAMP(EndTime)) as EarliestEndTime,
max(UNIX_TIMESTAMP(EndTime)) as LatestEndTime,
date_format(min(EndTime),"%m") as Month,
date_format(min(EndTime),"%Y") as Year,
IF(DistinguishedName NOT IN (\"\", \"Unknown\"),
LEFT(DistinguishedName,INSTR(DistinguishedName,\":/\")-1), DistinguishedName),CommonName) as GlobalUserName,
Round(Sum(WallDuration)/3600) as WallDuration,
Round(Sum(CpuUserDuration+CpuSystemDuration)/3600) as CpuDuration,
Round((Sum(WallDuration)/3600) * NORMALIZATION_FACTOR ) as NormalisedWallDuration,
Round((Sum(CpuUserDuration+CpuSystemDuration)/3600) * NORMALIZATION_FACTOR) as NormalisedCpuDuration,
Sum(NJobs) as NumberOfJobs
Site.SiteName in (LIST_OF_MYOSG_RESOURCES_WITH_INTEROPACCOUNTING_FLAG_TRUE)
and Site.siteid = Probe.siteid
and Probe.ProbeName = Main.ProbeName
and Main.VOName in ( LIST_OF_REPORTABLE_VOS )
and Main.ResourceType = "Batch"
group by Site,
The reason for the complexity in extracting DistinguishedName in Gratia is two-fold:
- the check for empty or 'Unknown' is because the DistinguishedName was not available in Gratia until 11/1/2009
- the even more complex exclusion of everything after the first ":/' was to compensate for a new feature in Condor 7.3.2 and 7.4 that captured the VOMS proxies extended attributes and appended them to the x509userproxysubject attribute which is the source of the data.
The filter on ResourceType was due to a change in the condor probes (v1.00.3) to distinguish between the actual grid jobs (Batch) and the grid monitoring job (GridMonitor) when jobs are submitted using condor_submit. Any 'local' job used to submit a job on the CE node will be filtered, but should they at some point be passed to Gratia, these will be identified as 'Local'.
There are several integrity checks that the LCG.py process makes to insure all data is being reported correctly.
Sites/resources are reporting.
As a part of the interface process, a check is made at the site/resource level (not individual probes) to determine if data is being reported to Gratia every day. If a site/resource has missing data for any day, checks are made to see if this is due to planned maintenance or if a site/resource has gone inactive based on OIM/MyOSG data. If after these checks are made and a site/resource still has missing data for more than 2 days in the month, an email "WARNING" notification is issued to initiate an OSG GOC ticket. Example:
Note: this is a manual ticket initiation.
WARNING/ADVISORY: Resource: UCSDT2-B in Resource Group: UCSDT2 missing data for more than 2 days: ['2014-03-19', '2014-03-20', '2014-03-21',]
If, for whatever reason MyOSG
is unavailable, an additional warning message will be generated advising of this. This is not a critical problem unless it persists for several. When this occurs, any messages regarding sites not reporting should not be acted upon.
- WARNING: Unable to determine planned shutdowns
Since, at the current time, only those OSG resource_groups
registered in the EGI REBUS topology
will be reported for MOU purposes, a validation is made to insure MyOSG
and the EGI REBUS topology
are in sync. The terminology between the two is slightly different:
| Federation Accounting Name
|| WLCGInformation Accounting Name
|| resource group
A query is made to EGI REBUS topology
and against MyOSG
comparing the MyOSG WLCGInformation AccountingName with the Rebus AccountingName to insure that monthly MOU reporting will be accurately for the resource_group's
data the interfaced sends to APEL. This should be investigated and a goc ticket should be generated to take correct the descrepancy. There are too many combination of conditions to highlight here. They are self explanatory as to the problem.
Note: this is a manual ticket initiation/process
The following warning can occur and are not immediately critical unless they persist:
- WARN: The WLCG REBUS topology was not accessible today. We are using a previous days data for validations
Other warning/error messages
On the first day of each month, a warning message will be generated advising that the following files should be added to svn:
- the current months lcg-reportableSites file
- These files should be checked to see if any updates should be made to SVN/CVS in order to retain their history.
Updating of the normalization factor for a resource_group
is a manual process.
The ALICE and CMS sites provide their own normalization factor.
ATLAS maintains their own calculations manually and can be viewed here:
A goc ticket should be generated to initiate a change.
When a change is made, svn should be updated for both the main /etc/gratia-apel/lcg-reportableSites
file and the appropriate /etc/gratia-apel/lcg-reportableSites.history
This is also a manual process. As seen in the section on warning messages, there is a process which checks MyOSG
at the resource
level for the InteropAccounting flag and warns if a site says it is reporting but has not yet been added to the lcg-reportableSites
A goc ticket should be generated to investigate.
Move the manually maintained configuration files to MyOsg
Eventually the lcg-reportableSites
file should be replaced with data in MyOSG
The areas that would have to be addressed are:
- Identification of the resource_groups that are to be reported/interfaced.
- Maintenance of the normalization factor at the resource_group level
- Identification of the VOs to be reported/interfaced
Synchronization of OIM/MyOSG with Gratia
The intent of this initiative is to align OIM/MyOSG with Gratia and, additionally, extend that to the APEL/WLCG interface. The motivation is to eliminate the manual intervention required when new resources (Gratia sites) are added or removed from service. Currently, there does not exist any true relationship between resources in OIM/MyOsg and Gratia. The OIM/MyOSG resource name should correspond to a Gratia site name. In the following paragraphs, when the term 'resource' is used, it is considered synonymous with Gratia 'site name'.
In OIM/MyOSG, the ability to define resources that are involved in the APEL/WLCG interface currently exists This data is at the resource level and contained in the WLCG Information section:
- Interop Accounting flag - True if the resource is interfaced to APEL/WLCG for accounting purposes
- Accounting Name - if the Interop Accounting flag is True, this contains the Tier 1 or 2 WLCG federation name (e.g., T2-US-Caltech)
From an APEL/WLCG perspective, we need to handle the following use cases:
- Addition of a new resource for a Tier1/2 Federation
- Removal of a resource from a Tier1/2 Federation (aka, disabling a resource in OIM/MyOSG terms). This requires preserving the historical data for that resource.
Validation/verification of monthly MOU reports
When the Tier 2 monthly reports are issued by the LCG accounting office (e.g, http://lcg.web.cern.ch/LCG/accounting/Tier-2/2010/january-10/Tier2_Accounting_Report_January2010.pdf
), a validation of the OSG federation resource is being performed manually. It would be "nice" to be able to able perform some form of automated validation and eliminate this manual effort. I am not sure if this is totally viable.
Normalization factor issues
- As far as Gratia/Accounting is concerned, we would like to get to a point where each job usage record contain the type of cpu that the job ran on. This would give Gratia the most flexibility to normalize the result (normalization is required by WLCG and is the only way to fairly compare the result).
However, at this point the batch system does not gives us enough information (Condor 6.9 might be able to make the information available). So for now we will rely for normalization on a rougher estimate. Currently the estimate comes from the average on one existing farm for which I know the actual hardware. When the GLUE schema is properly filled, we will use the average for the Site the job ran on.
The normalization factor is currently reflecting the latest cluster configuration. As these clusters are upgraded over time, these factors will change. In order to be able to re-create the normalized view of past periods, these configurations/factors will need to be keyed by time period somehow.
This section just documents when major changes to the interface took place:
| September 2008
|| Added methods to create xml/hmtl/dat files to provide visibility to the data sent and displayed at Gratia-APEL WLCG Interface and OSG WLCG Reporting
| November 2008
|| Added an additional filter on the selection criterea. Only selecting ResourceType='Batch'. Changes to the condor probes (v1.00.3) will result in distinguishing between the actual grid jobs (Batch) and the grid monitoring job (GridMonitor) when jobs are submitted using condor_submit. Any 'local' job used to submit a job on the CE node will be filtered, but should they at some point be passed to Gratia, these will be identified as Local.
| March 2009
|| Started saving the configuration file for reportable sites and normalization factors for each month. This provided the capability to re-send prior months data in the event a site had problems and still retain that period's configuration.
| May 2009
|| Added the User's Distinguished Name (DN) as part of the summary criterea and data sent.
| June 2009
|| Started checking for Gratia sites that had stopped reporting data to Gratia. This provided a pro-active means of identifying problems rather than waiting for the monthly MOU reports to be issued.
| July 2009
|| Added an extract of OIM downtimes to use when determining if a Gratia site failed to report
| December 2009
|| Added use of a new class (InactiveResources) to query MyOsg? for Resource that have been marked inactive. This is used in conjunction with the Downtimes class when checking why a site/resourcehas not reported
| January 2010
|| APEL/EGI started reporting MOU reports using HEPSpec2006 normalization factors in place of the SpecInt2000 values we are reporting. We still use SpecInt2000. They apply a factor of 4x as means of adjusting until all sites report using HEPSpec2006. When we have HEPSpec2006 available we divide by 4 to maintain consistency.
| December 2010
|| The ALICE VO was added to the list of reportable VOs and sites.
| August 2011
|| This represents a major change to the interface. All OSG reporting to WLCG has been in reference to MyOsg resource group. However, the MyOsg InteropAccounting flag for WLCG Information is at the resource, not resource group, level. The changes made in this revision will now:
1. treat the lcg-reportableSites config file entries as resource groups. This also means the Normalization Factors should be calculated at the resource group level.
2. determine which resources within that resource group should have their gratia data reported to APEL/EGI.
3. summarize (using sql) the accounting data for the month for all interfaced resources for the resource group. A new class, InteropAccounting(.py), provides the access to MyOsg.
| June 2012
|| Changed the APEL interface from a direct update of the APEL database to the use of a messaging system (SSM)
More detail information related to code, reportable sites and normalization factors can be found in the Gratia Interfaces - APEL/WLCG - History twiki
Sending Data to EGI Based on VO Only
APEL SSM/ActiveMQ Interface
- 19 Nov 2009: Updated for implementation of User Distinguished Name on 11/1/2009
- 08 Feb 2010: Split the developers corner (doc) out as a separate twiki (ApelWlcgDevelopersCorner)