You are here: TWiki > SoftwareTools Web>PlansTimelines>CommonTestFormat (03 Sep 2010, AlainRoy?)
Draft Page

Next Meeting (Friday August 30, 10:30 Central Time)

  • 605-475-4800 883253#

Requirements

I believe we have a consistent set of requirements:

  • Status with Pass/Fail
  • A list of key/value pairs
  • Optional debugging output

June 28th, Igor wrote:

So the data you want is similar to what I (glideinWMS) am interested in:
a) Pass/Fail,
b) list of (key,value) pairs

PS: Just don't call it CONDOR_AD:*... should be framework neutral.

June 28th, Mats Ryne wrote:

We *need* debugging output. Yes, the tests are going to be used
*mostly* in automated fashion, but once they fail we need information.
If the tests do not have enough debugging output, we will probably
write our own tests again just to get that information.

June 28th, Doug Strain wrote: We'd like:

- Test status:  Preferably not only PASS/FAIL, but a general string, such as Warning, 
  Not run, Not available, or other status 
  (though I think it would be sufficient to push this to a different field if necessary)
- Human-readable Description or an error code, for further information
- A long description, with details, like stdout/stderr of the commands run, logfile, etc.
- Hostname of target, as well as the hostname of the source (where the test is run), as well as a srm url.
- Optional extensible fields (could be key/value pairs) to return other information.  Ie, for a "srmping" test, return the backendtype (dcache, bestman, etc) if it is given.

As RSV comes very close to solving our issues and we already use it for other probes, this format suits us well.
I think we are also amenable to other formats if it is accepted as standard.

Desires

  • [Igor] It would be nice to be easy to process with command-line tools. (We refer to this colloquially as "greppable" output)
  • [Maxim, Robert] It would be nice to have have a standard format (XML, JSON) for which there exist good, well supported and documented parsers.
  • [Alain] It would be nice to be able to easily re-use these tests within RSV if we wish.
  • [Alain] It would be nice if the output format was easily convertible to other formats to assist with these. (For instance, if we have XML format, it would be nice to convert it to RSV. Or vice-versa for the benefit of people that want XML.

References

Examples

Sample test output from one of Derek Weitzel's experiments:

metricName: org.osg.wn.osgdata-available
timestamp: 2010-07-02T15:05:29.0CDT
metricType: status
metricStatus: PASSED
detailsData: 
EOT

The equivalent in JSON, by Maxim:

{
  "name": "org.osg.wn.osgdata-available",
  "time": "2010-07-02T15:05:29.0CDT ",
  "type": "status",
  "result": "PASSED",
  "detailsData": ""
}

A possible equivalent in XML (by Igor):

<OSGTestResult>
  <description>
   <property name="id" value="org.osg.wn.osgdata.probe"/>
   <property name="started" value="2010-07-02T15:05:29-05:00"/>
  </description>
  <result>
   <status value="OK"/>
   <metric name="org.osg.wn.osgdata-available" collected="2010-07-02T15:05:29-05:00" uri="local" value="PASSED"/>
   <detail></detail>
  </result>
</OSGTestResult>

Current proposals

  • Use XML. One test result can contain multiple metrics, which with a different result.
  • Each element should be on a separate line for ease of cmdline parsing. (See example below) But note: This is not a hard requirement because tools like xmlline --format can be used when necessary. However, it is recommended for purposes of readability.
  • There are some mandatory fields:
    • test unique name
    • timestamp: ISO8601
    • test result
  • Optional fields are possible:
    • metrics results
  • While this stage of discussions and design concentrates on the format of a single test entry, we envisage serializing results of a few test in a single document, with an appropriate collection tag.

Other constraints:

  • None yet

Igor's proposal

XML example:

<OSGTestResult>
  <description>
   <property name="id" value="org.osg.wn.glexec.probe"/>
   <property name="version" value="1.2"/>
   <property name="command" value="/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data"/>
   <property name="started" value="2010-07-02T15:05:00-05:00"/>
   <property name="ended" value="2010-07-02T15:08:22-05:00"/>
  </description>
  <result>
   <status value="OK"/>
   <metric name="org.osg.wn.glexec.location" collected="2010-07-02T15:05:29-05:00" uri="local" value="/usr/bin/glexec"/>
   <metric name="org.osg.wn.glexec.version" collected="2010-07-02T15:05:29-05:00" uri="local" value="1.2.3.foo"/>
   <metric name="org.osg.wn.glexec.GUMS.version" collected="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums" value="1.3.1"/>
   <detail>Discovered glexec binary in /usr/bin.
   Version appears to be 1.2.3.foo
   glexec mapping for pilot proxy PASSED.
   </detail>
  </result>
</OSGTestResult>

Maxim's proposal (Version 1)

The examples below are based upon most accessible sources, about best practices in XML design, -- however, it's true that this is a "grey area" and there is no definitive "best". It is possible that this format has better readability than the above example, and is still reasonably easy to parse with Unix filters if needed. The format of the "result" section is an example of a compromise, to make it so. The "detail" was put it in the same level as other sections (then, the "result" is just a collection of metrics and this is simpler). Detail characterizes the outcome of the whole test.

<OSGTestResult id="org.osg.wn.glexec.probe" version="1.2">
  <test>
     <cmd>/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</cmd>
     <tStart>2010-07-02T15:05:29-05:00</tStart>
     <tEnd>2010-07-02T15:05:29-05:01</tEnd>
  </test>
  <result>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
     <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
     <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

Maxim's Proposal (Version 2)

<OSGTestResult>
  <id version="1.2">org.osg.wn.glexec.probe</id>
  <test tStart="2010-07-02T15:05:29-05:00" tEnd="2010-07-02T15:05:29-05:01">/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</test>
  <result>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
     <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
     <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

Maxim's proposal (Version 3)

More info added based on Robert's suggestions.

<OSGTestResult id="org.osg.wn.glexec.probe" version="1.2">
  <OperatingEnvironment>
     <site>BNL</site>
     <hostname>acas0059</hostname>
     <cwd>/tmp</cwd>
     <OS>SL5</OS>
  </OperatingEnvironment>
  <test>
     <cmd>/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</cmd>
     <tStart>2010-07-02T15:05:29-05:00</tStart>
     <tEnd>2010-07-02T15:05:29-05:01</tEnd>
  </test>
  <result>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
     <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
     <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

Alain's proposal (Version 4, evolution of Version 3)

The XML output has two possible structures.

  1. A single test result, encapsulated with the <OSGTestResult> tag.
  2. A set of one or more test results (the same <OSGTestResult> element), encapsulated by a <OSGTestResultSet> tag

We recommend that tests output the former when applicable (to simplify parsing on the command-line), but the latter can be used when necessary. In the case where there is a set of results, they are considered to be independent tests, each with their own result. That is, it is the same as having somehow individually obtained multiple <OSGTestResult>'s.

A single test result, <OSGTestResult>, contains four parts:

The Operating Environment
The operating environment is an extensible set of attributes to describe the environment in which the test ran. Because collection of this data may be difficult or ill-defined in some circumstances, this section is not required, and if included, any of the attributes included or omitted. We label each of these attributes as recommended (it should be included when it's feasible) or optional (include at your discretion).

Name Kind Comment
sitename Recommended In an OSG job environment, this is OSG_SITE_NAME.
May not make sense outside of OSG.
hostname Recommended Should be fully-qualified hostname of primary network interface (equivalent to hostname -f).
May not make sense for computers not on public network
cwd Optional The current working directory for the job
os Optional The operating system for the computer on which the test was run.
This is non-standard, human readable text, but we recommend using the Description field fro the lsb_release command when available.
arch Optional The CPU architecture. Common values: i686, x86_86.
This is non-standard, human readable text, but we recommend using the output of uname -m when available.
user Optional The local user name that ran the test, such as "roy"
userdn Optional The distinguished name from the certificate of the user that ran the test, such as "/DC=org/DC=doegrids/OU=People/CN=Alain Roy 424511"
other Optional The test may include any other attributes as appropriate

The test
The precise test that was run, and the time interval during which it ran. This section is optional, but if it's specified, the cmd must be specified. There are three attributes, all of which must be specified.

Name Kind Comment
cmd Required The command-line that was invoked to run the test.
tStart Recommended The time the command began.
tEnd Recommended The time the command ended.

The results
The overall result of the test as well as the specific findings of the test. This section is required.

Name Kind Comment
status Required The result of the status. Must be one of OK, FAILED, WARNING, or UNKNOWN.
metric Recommended A specific test result. If used, must include three XML attributes: name, ts (timestamp), and uri (resource on which it was run, can be "local")

Detail
Human-readable commentary on the test. In general, may not be machine-parseable.

Non-normative example for a single test result

<?xml version="1.0"?>
<OSGTestResult id="org.osg.wn.glexec.probe" version="1.2">
  <operatingenvironment>
     <env name="sitename">WISC-OSG-EDU</env>
     <env name="hostname">osg-edu.cs.wisc.edu</env>
     <env name="os">Scientific Linux 5.5</env>
     <env name="numcpus">4</env> <!-- An example of extending the list of attributes, based on what's useful to the test author -->
  </operatingenvironment>
  <test>
     <cmd>/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</cmd>
     <tStart>2010-07-02T15:05:29-05:00</tStart>
     <tEnd>2010-07-02T15:05:29-05:01</tEnd>
  </test>
  <result>
     <status>OK</status>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
     <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
     <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

Non-normative example for a multiple test results

<?xml version="1.0"?>
<OSGTestResultSet>
<OSGTestResult id="org.osg.wn.glexec.probe" version="1.2">
  <operatingenvironment>
     <env name="sitename">WISC-OSG-EDU</env>
     <env name="hostname">osg-edu.cs.wisc.edu</env>
  </operatingenvironment>
  <test>
     <cmd>/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/test1.pl /osg/data</cmd>
  </test>
  <result>
     <status>OK</status>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

<OSGTestResult id="org.osg.wn.gridproxyinit.probe" version="1.2">
  <operatingenvironment>
     <env name="sitename">WISC-OSG-EDU</env>
     <env name="hostname">osg-edu.cs.wisc.edu</env>
  </operatingenvironment>
  <test>
     <cmd>/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/test2.pl /osg/data</cmd>
  </test>
  <result>
     <status>OK</status>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/grid-proxy-init</metric>
  </result>
</OSGTestResult>
</OSGTestResultSet>

Maxim's Parser demo

Input

We chose of a variation of "Version 2" example above as input for a parser based on Python minidom. There is more than one OSGTestResult in the document, and so the tag of the collection was named OSGTestSuite (provisionally and open to discussions).


<?xml version="1.0"?>
<OSGTestSuite>
   <OSGTestResult>
      <id version="1.2">org.osg.wn.glexec.probe</id>
      <test tStart="2010-07-02T15:05:29-05:00" tEnd="2010-07-02T15:05:29-05:00">/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</test>
      <result>
         <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
         <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
         <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
      </result>
      <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
   </OSGTestResult>
   <OSGTestResult>
      <id version="1.0">org.osg.wn.pwd</id>
      <test tStart="2010-07-02T15:05:40-05:00" tEnd="2010-07-02T15:05:41-05:00">/bin/pwd</test>
      <result>
         <metric name="pwd" ts="2010-07-02T15:05:40-05:00" uri="local">/u/tmp_a5007</metric>
      </result>
   </OSGTestResult>
</OSGTestSuite>

Python code

Python code that parses documents in the above format:

from xml.dom.minidom import parse, parseString
#...
testDom = parse('/tmp/'+filename)
     
txt=''
for tr in testDom.getElementsByTagName('OSGTestResult'):
    e=tr.getElementsByTagName('id')[0]
    txt+='id: '+e.firstChild.nodeValue+' version:'+e.getAttribute("version")+'<br/>'

    e=tr.getElementsByTagName('test')[0]
    txt+='test: "'+e.firstChild.nodeValue+'"     tStart:'+e.getAttribute("tStart")+' tEnd:'+e.getAttribute("tEnd")+'<br/>'

    txt+='Metrics:<br/>*** *** ***<br/>'
    for m in tr.getElementsByTagName('result')[0].getElementsByTagName('metric'):
        attrs=m._get_attributes()
        for key in attrs.keys():
            txt+= key+':'+ attrs[key].nodeValue+'<br/>'

        txt+='Result: '+m.firstChild.nodeValue+'<br/>*** *** ***<br/>'
    txt+='<hr/>'
print txt

Output

id: org.osg.wn.glexec.probe version:1.2
test: "/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data"     tStart:2010-07-02T15:05:29-05:00 tEnd:2010-07-02T15:05:29-05:00
Metrics:
*** *** ***
uri:local
name:location
ts:2010-07-02T15:05:33-05:00
Result: /usr/bin/glexec
*** *** ***
uri:local
name:version
ts:2010-07-02T15:05:33-05:00
Result: 1.2.3foo
*** *** ***
uri:https://gums.fnal.gov/gums
name:GumsVer
ts:2010-07-02T15:05:33-05:00
Result: 1.3.1
*** *** ***
_____________________________________________________
id: org.osg.wn.pwd version:1.0
test: "/bin/pwd"     tStart:2010-07-02T15:05:40-05:00 tEnd:2010-07-02T15:05:41-05:00
Metrics:
*** *** ***
uri:local
name:pwd
ts:2010-07-02T15:05:40-05:00
Result: /u/tmp_a5007
*** *** ***
_____________________________________________________

This parser demo was installed as a part of a Django application, which makes it easy to experiment with test data persistence in the database, if needed.

Robert's Proposal

My proposal is based on Maxim's Version 1.

  • move start and end time to be attributes to cmd
  • add the current working directory of the test cwd as a new attribute to cmd
  • add entity user with attributes dn and local
  • add entity host with attributes name , fqdn and identifier

<OSGTestResult id="org.osg.wn.glexec.probe" version="1.2">
  <test>
     <cmd cwd="/home/ligo" start="2010-07-02T15:05:29-05:00" end="2010-07-02T15:05:29-05:00">/usr/bin/perl /raid2/osg-data/osg/ttk/filesystem/permissions.pl /osg/data</cmd>
     <user dn="/DC=org/DC=doegrids/OU=People/CN=Robert Engel 392994" local="uid=506(ligo) gid=506(ligo) groups=506(ligo)"</user>
     <host name="node321" fqdn="node321.ligo.caltech.edu" identifier="LIGO_CIT">
  </test>
  <result>
     <metric name="location" ts="2010-07-02T15:05:33-05:00" uri="local">/usr/bin/glexec</metric>
     <metric name="version"  ts="2010-07-02T15:05:33-05:00" uri="local">1.2.3foo</metric>
     <metric name="GumsVer"  ts="2010-07-02T15:05:33-05:00" uri="https://gums.fnal.gov/gums">1.3.1</metric>
  </result>
  <detail>Discovered glexec binary in /usr/bin. Version appears to be 1.2.3.foo. glexec mapping for pilot proxy PASSED</detail>
</OSGTestResult>

Open Questions

  • Should we have a standard for naming our metrics, similar to the WLCG probe output? (Such as Igor's example of "org.osg.wn.glexec"?)

    Igor
    Maybe for the OSG standard tests, but VO provided ones should have no restrictions.
  • Should we declare standard command-line parameters for our tests, similar to the WLCG probe output?

    Igor
    Yes... but that should be a different thread.
Topic revision: r22 - 03 Sep 2010 - 17:33:42 - AlainRoy?
Hello, TWikiGuest
Register

 
TWIKI.NET

TWiki | Report Bugs | Privacy Policy

This site is powered by the TWiki collaboration platformCopyright by the contributing authors. All material on this collaboration platform is the property of the contributing authors..