xdrlib

Thinking about how to shuffle log events between Python and other languages (e.g. Java), XDR seems like one possible solution. It would allow non-string data types (i.e. encode the timestamps as floats) and minimize parsing on the receiver. So I fooled around a little with python’s xdrlib. Pretty nice, although it assumes an RPC model of one message at a time. I saw an interesting proposal to add a streaming read() function on the Python bug tracker, but that seems to me like overkill (especially since I intend to do my reading from Java).

Instead, I used a simple buffer length header. Here are the write_buf(), read_buf() functions. I’m using files here, but sockets should be the same (except for the usual issues with read() not returning as many bytes as you want).

def write_buf(ofile, p):
    """Write the contents of xdrlib.Packer, p,
    to ofile with a 4-byte header with the buffer size.
    """
    buf = p.get_buffer()
    buflen = len(buf)
    p.reset()
    p.pack_int(buflen)
    ofile.write(p.get_buffer())
    ofile.write(buf)
    return buflen

def read_buf(ifile, p):
    """Read a buffer from ifile into the xdrlib.Unpacker, p,
    using the 4-byte header to determine the data buffer size.
    """
    hdr = ifile.read(4)
    if not hdr:
        return 0
    p.reset(hdr)
    bufsize = p.unpack_int()
    data = ifile.read(bufsize)
    p.reset(data)
    return bufsize

pylint

Although I’ve used pychecker for a while to speed up the Python development process, I never got into pylint. But I think it’s a good time now to go through and modernize & streamline the Python codebase a bit. So I’ll be posting pylint usage notes here.

First things first: to get pylint, go to the pylint home page and install it in the usual way.

Then after you set up PYTHONPATH as you would to import a module, you can run it on code. So far, I have only used it in the most basic way

• Check errors: pylint -e {module}.py
• Full report: pylint {module}.py

I understand there’s a great deal of customization possible, but I’m going to live with the defaults for a while and see where that gets me. Some things in my code that in general need to be changed are:

• methods named doSomething() should be renamed do_something(). I like this convention better anyways.
• I use a global ‘log’ object; pylint wants globals like that to be capitalized. I tried it and kinda like it. So, log.info(”foo”) becomes LOG.info(”foo”)
• Short variable names generate warnings. This is particularly evident in exception handling, where I use the variable ‘E’ for the exception object. I think it is fine to change this to ‘err’.
• Undocumented methods generate warnings. No problem there.
• Instance attributes initialized outside the constructor. This is just bad practice, so thanks pylint.

Look for the ‘pylint’ tag for updates

NCSA / OSG trial deployment of NetLogger Pipeline

Over the past few days, Jim Basney and I worked through the, currently manual, steps of getting a pipeline running collecting info about his test OSG cluster.  The idea is to have NetLogger become a VDT package for site admins to monitor the security of their site.

There were a few small hiccups but for the most part it went well.  Jim wrote it up here.  Now, we’ve got some bugs to fix and perhaps a feature or two to add.

Connecting to R from Java

Background: NetLogger uses R for analysis, but the Pegasus group has a Java GUI. They want to re-use the R code directly by connecting to R from Java.

I think this project (JRI) has what’s needed:

home: http://www.rforge.net/JRI/
download page: http://www.rforge.net/JRI/files/
javadoc page:http://www.rosuda.org/R/nightly/JavaDoc/

It compiled and seemed to work on Mac OSX (Jaguar, with Developer tools installed).

If we just want graphs from R, the basic interface is all we’d need. Here’s an example of how I ran a little test program (below) from inside the unzipped tarball. When it finishes you can look at the output in “/tmp/nlr.pdf”.


JAVA_HOME=/Library/Java/Home
R_HOME=/Library/Frameworks/R.framework/Resources
javac -classpath src/JRI.jar examples/nlr.java
java -cp src/JRI.jar:.:examples nlr
(Output)
creating Rengine
Rengine created, waiting for R
R is running
stopping Rengine


Test program:


import org.rosuda.JRI.Rengine;
import org.rosuda.JRI.RMainLoopCallbacks;

return;
}
}


updated bestman (SRM) parser and R code

I’ve updated the bestman parser, which was previously trying to link together the events by a combination of path and thread-id, to do a much simpler operation using the request identifier (rid). This is a big improvement over the previous version linked to here, which wasn’t trying to link them together at all (!) In addition to making the code simpler and more reliable, it also should help decoding the raw output as the rid has some information about the user and type of request inside it.

The new code is in subversion and should be in the development release tomorrow (Dec 12). I also updated the R code to use the new name in the ‘ident’ table (’req’ instead of ‘guid’). See the previous post for details on getting the code out of subversion, running the parser and loader, etc.

You can get a new sample logfile off the NetLogger Wiki on the SRM/BeStMan page.

parsing SRM logs

I’ve updated the SRM parser in an attempt to more easily link together the various events with a unique identifier (which I’m putting in the ‘guid’ attribute).

To get the newest code, follow the cookbook instructions for getting the code from subversion and setting up your environment.

Then, you can test out the parser on this sample log. After saving the log to some file we’ll just call $LOGFILE, do this:

nl_parser -m bestman -p version=2 $LOGFILE


This should print, to standard output, 14 lines of pretty unreadable log output. However, if you save this in a file, say $LOGFILE2, you could now load it into a database with nl_loader. Here’s an example of how to load it into an sqlite3 database (file) called $DBFILE — no extra installation is required for SQLite.

nl_parser -m bestman -p version=2 $LOGFILE > $LOGFILE2
DBFILE=example.sqlite
nl_loader -u sqlite://$DBFILE -C -i $LOGFILE2


To verify that there actually is something in the database, you can do a simple query like this:
sqlite3 $DBFILE "select * from event"
# Output:
1|207deecdec2f9a3f0c8d33ccd7298072|1225524876.722|srm.server.copy.in|2|4
2|0b468528bc886b71176a1f4bb1f8a0a4|1225524876.731|srm.server.list|2|4
3|106f4156e5517858ec0f4684dfe73a9d|1225524876.731|srm.server.req|0|4
4|597e7ba2a8cb6383cc871e383d05e350|1225524876.732|srm.server.req.to.queued|2|4
5|82973ccd386f6cee9ebf5fe33ef71e69|1225524876.732|srm.server.req.to.scheduled|2|4
6|f8fa6278a718101c9a4a4482744f5548|1225524876.733|srm.server.copy.out|2|4
7|0a50c7178490b8f8904e52ea863afd7c|1225524876.894|srm.server.req.to.status|2|4
8|014e3fef040a8e048d09e059ae7987b4|1225524876.894|srm.server.TSRMRequestCopyToRemote.upload|2|4
9|ab2bb06914ce590266bca5117c11ed91|1225524876.894|srm.server.req.to.status|2|4
10|d16c585253b2e420be8322fc64f759fc|1225524876.895|srm.server.tx.push|0|4
11|7b7370e35a8917ae5f976dc6f4f611f9|1225524876.895|srm.server.tx.push.size|2|4
12|dc1d0b07b1b404be2be3c2a7ddf23148|1225524935.733|srm.server.tx.push|1|4
13|ee8bc3e776910fae76f3ec310eaba879|1225524935.733|srm.server.req.to.status|2|4
14|d2c23970199473453820ac07aec5eeb7|1225524935.733|srm.server.req.to|1|4


You could also try out the R code in trunk/R/bestman.R in the NetLogger subversion repository. Just looking at the contained SQL should give you an idea how this data can be queried and joined together on those ‘guid’ attributes so you can look at all the information from one transfer together. Note that there are still some issues with getting the same GUID for all the events, and in particular the ‘queued’ event doesn’t seem to get the same GUID as the rest.

If you want to look at the source code for the parser itself, look under trunk/python/netlogger/parsers/modules/bestman.py and in particular the code section that mentions “version 2″. Hopefully it’s reasonably clear how events are being mapped and processed.

Script to indent R code

I was looking for a way to indent my R code, since the R auto-packaging command inserted a bunch of hard tabs and anyways the code was messy.

I found this page describing a way to do it with ESS (Emacs Speaks Statistics). So, I downloaded ESS from the ESS homepage and adapted the script slightly to be a simple shell-script.

#!/bin/sh
# Use emacs ESS pkg to indent R file
# Dan Gunter, Dec 2008
function usage () {
printf "Indent R file with Emacs ESS package.\n"
printf "Usage: $0 FILE\n"
exit 1
}
f=$1
shift
if test "x$f" = x -o "x$f" = "x-h"; then
usage
fi
emacs -batch \
-eval '(load "/Users/dang/local/lib/emacs/ess-5.3.8/lisp/ess-site")' \
-f R-mode \
-eval '(insert-file "'${f}'")' \
-eval '(set-visited-file-name "'"${f}"'")' \
-eval '(indent-region (point-min) (point-max) nil)' \
-f save-buffer \
2>/dev/null


packaging R scripts

One of the tedious sub-tasks necessary for distributing the analysis code I use is to package up the R programs (not really “scripts” as it’s a full-featured language) into something that can be run from the command-line.

R does not have the best interface to this, but I found online at a script to run R programs using #! syntax that uses the R startup sequence to “fool” the interpreter into running your script as part of its initial commands. I tweaked this a bit, to use printf instead of echo and to automatically quit the interpreter after the called script returns, and then checked it into subversion as “runR” in the trunk/R directory.

But how to take my current files of various functions and make them into a “real” executable? What I really need is the equivalent of Python’s optparse, or at least getopt. Time to do some more hunting..

Poor man’s pipeline

As a short-cut to getting a full nl_pipeline running (because I suck) here’s what I just did to load in from scratch all the SRM logs and start a “poor man’s pipeline” to pick up events as they come in:

# Start with a completely clean DB
mysql> drop database test;
mysql> create database test;

# Load in existing events from the existing log file (note the -C passed to nl_loader to create the schema)
$ grep " event_srm_log " pdsfgrid5.nersc.gov.vdt.log | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -C -u mysql://localhost -p db=test
# This took about 35 minutes to complete, yielding some 220K events in the DB

# Now the pipeline:
$ tail -f pdsfgrid5.nersc.gov.vdt.log | \
  grep " event_srm_log " | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -u mysql://localhost -p db=test

I’ve also done this for gram events for the tech-x guys…

SQL code to get GRAM job ID from URL


mysql> use techx;
mysql> select trim(trailing '.' from replace(substring_index(gridJobID, '/', -3), '/', '.')) as 'gram_id' from Jobs;

+---------------------+
| gram_id |
+---------------------+
| 14094.1226615002 |
| NULL |
| 16255.1226616087 |
| 19674.1226619065 |
| 22903.1226620711 |
| 28114.1226625017 |
| 29889.1226625489 |
| NULL |
| 11191.1226683572 |
| 25930.1226693279 |
| 30603.1226697028 |
| 7016.1226699246 |
+-------------------- -+
12 rows in set (0.00 sec)