Progress: July 2003 to October 2003
The focus of the DMF project for this quarter has been to work with the GGF to arrive at a common schema for network monitoring, to enhance the NetLogger "destination URL" and trigger mechanism, and to develop a prototype "continuous query" component for NetLogger messages, and to work with Grid3/PPDG to add NetLogger to the Grid3 effort. More details on each of these topics follow.
The work for this quarter
has been split between NetLogger and XML Schemas for OGSI integration. For
NetLogger, we focused on better NetLogger documentation and packaging, and
some enhancements. The destination for NetLogger messages, which has long
been conceptually a URL (Universal Resource Locator) finally used some of
the generality of URL's to embed directives about the message format, logging
level, and constant values into the URL itself. This exposes many of NetLogger's
features in a manner more convenient for job execution scripts and Grid middleware.
We have also developed a kinder and gentler format for NetLogger "trigger
files", which can redirect monitoring data in a running application, so that
they can be trivially modified with any text editor. We also continued work
on a first prototype to perform "continuous", i.e. streaming real-time, relational
queries over NetLogger data streams, which promises to add powerful real-time
filtering to NetLogger. We also greatly improved the documentation for NetLogger.
On the XML Schema / OGSI front, we are also leading the development an XML Schema for all Grid network measurements within the GGF Network Measurement working group. As this implies, we continue to be extremely active in the Global Grid Forum. Brian and Dan attended GGF in Chicago, where Brian co-led a Network Measurement working group all-day F2F meeting as well as a working group session. The XML Schema work follows partly from the earlier NMWG document to define a CIM profile for network measurement data. Brian Tierney concluded his duties as the chair of the GGF Nominating Committee by announcing the newly elected members of the GGF steering committee at the opening plenary session.
For the iVDGL/PPDG Grid3 effort, we created a NetLogger repository service for use by all of Grid3. See http://www-didc.lbl.gov/NetLogger/netlogger-grid3.html. We also built a NetLogger-instrumented version of globus-url-copy using pyGlobus, and deployed this on all the Grid3 sites. We are using this to monitoring the status of Grid3 sites.
We started implementing a Grid Troubleshooting analysis demo with the DOE Science Grid for SC2003. This involves adding or standardizing the use of NetLogger in several Grid components including Globus, pyGlobus, and Akenti.
We completed the final version of the paper "Service "On-Demand Grid Application Tuning and Debugging with the NetLogger Activation Service ", which we submitted to the 4th International Workshop on Grid Computing (Grid2003).
We prepared for the DMF project review, which took place at Argonne National Lab in August.
We continue to collaborate
with several groups, including NLANR, EU DataGrid, Globus, and the IEPM project
at SLAC on the possible use of NetLogger
to collect monitoring data for their projects. We are begging to investigate
the possibility of collaborating with the "Telegraph Continuous-Query" group
at UC Berkeley, led by Joseph Hellerstein, to add a streaming query capability
to NetLogger.
We continue working with several groups to help add NetLogger instrumentation to their software. This quarter this includes Bill Allcock and Jenny (ANL), Constantinos Dovorlis (GA Tech), IEPM (SLAC), EU DataGrid R-GMA, and Atlas Athena software.