Instead, I used a simple buffer length header. Here are the write_buf(), read_buf() functions. I’m using files here, but sockets should be the same (except for the usual issues with read() not returning as many bytes as you want).

def write_buf(ofile, p):
    """Write the contents of xdrlib.Packer, p,
    to ofile with a 4-byte header with the buffer size.
    """
    buf = p.get_buffer()
    buflen = len(buf)
    p.reset()
    p.pack_int(buflen)
    ofile.write(p.get_buffer())
    ofile.write(buf)
    return buflen

def read_buf(ifile, p):
    """Read a buffer from ifile into the xdrlib.Unpacker, p,
    using the 4-byte header to determine the data buffer size.
    """
    hdr = ifile.read(4)
    if not hdr:
        return 0
    p.reset(hdr)
    bufsize = p.unpack_int()
    data = ifile.read(bufsize)
    p.reset(data)
    return bufsize

First things first: to get pylint, go to the pylint home page and install it in the usual way.

Then after you set up PYTHONPATH as you would to import a module, you can run it on code. So far, I have only used it in the most basic way

• Check errors: pylint -e {module}.py
• Full report: pylint {module}.py

I understand there’s a great deal of customization possible, but I’m going to live with the defaults for a while and see where that gets me. Some things in my code that in general need to be changed are:

• methods named doSomething() should be renamed do_something(). I like this convention better anyways.
• I use a global ‘log’ object; pylint wants globals like that to be capitalized. I tried it and kinda like it. So, log.info(”foo”) becomes LOG.info(”foo”)
• Short variable names generate warnings. This is particularly evident in exception handling, where I use the variable ‘E’ for the exception object. I think it is fine to change this to ‘err’.
• Undocumented methods generate warnings. No problem there.
• Instance attributes initialized outside the constructor. This is just bad practice, so thanks pylint.

Look for the ‘pylint’ tag for updates

There were a few small hiccups but for the most part it went well.  Jim wrote it up here.  Now, we’ve got some bugs to fix and perhaps a feature or two to add.

I think this project (JRI) has what’s needed:

home: http://www.rforge.net/JRI/
download page: http://www.rforge.net/JRI/files/
javadoc page:http://www.rosuda.org/R/nightly/JavaDoc/

It compiled and seemed to work on Mac OSX (Jaguar, with Developer tools installed).

If we just want graphs from R, the basic interface is all we’d need. Here’s an example of how I ran a little test program (below) from inside the unzipped tarball. When it finishes you can look at the output in “/tmp/nlr.pdf”.


JAVA_HOME=/Library/Java/Home
R_HOME=/Library/Frameworks/R.framework/Resources
javac -classpath src/JRI.jar examples/nlr.java
java -cp src/JRI.jar:.:examples nlr
(Output)
creating Rengine
Rengine created, waiting for R
R is running
stopping Rengine


Test program:


import org.rosuda.JRI.Rengine;
import org.rosuda.JRI.RMainLoopCallbacks;

return;
}
}


The new code is in subversion and should be in the development release tomorrow (Dec 12). I also updated the R code to use the new name in the ‘ident’ table (’req’ instead of ‘guid’). See the previous post for details on getting the code out of subversion, running the parser and loader, etc.

You can get a new sample logfile off the NetLogger Wiki on the SRM/BeStMan page.

To get the newest code, follow the cookbook instructions for getting the code from subversion and setting up your environment.

Then, you can test out the parser on this sample log. After saving the log to some file we’ll just call $LOGFILE, do this:

nl_parser -m bestman -p version=2 $LOGFILE


This should print, to standard output, 14 lines of pretty unreadable log output. However, if you save this in a file, say $LOGFILE2, you could now load it into a database with nl_loader. Here’s an example of how to load it into an sqlite3 database (file) called $DBFILE — no extra installation is required for SQLite.

nl_parser -m bestman -p version=2 $LOGFILE > $LOGFILE2
DBFILE=example.sqlite
nl_loader -u sqlite://$DBFILE -C -i $LOGFILE2


To verify that there actually is something in the database, you can do a simple query like this:
sqlite3 $DBFILE "select * from event"
# Output:
1|207deecdec2f9a3f0c8d33ccd7298072|1225524876.722|srm.server.copy.in|2|4
2|0b468528bc886b71176a1f4bb1f8a0a4|1225524876.731|srm.server.list|2|4
3|106f4156e5517858ec0f4684dfe73a9d|1225524876.731|srm.server.req|0|4
4|597e7ba2a8cb6383cc871e383d05e350|1225524876.732|srm.server.req.to.queued|2|4
5|82973ccd386f6cee9ebf5fe33ef71e69|1225524876.732|srm.server.req.to.scheduled|2|4
6|f8fa6278a718101c9a4a4482744f5548|1225524876.733|srm.server.copy.out|2|4
7|0a50c7178490b8f8904e52ea863afd7c|1225524876.894|srm.server.req.to.status|2|4
8|014e3fef040a8e048d09e059ae7987b4|1225524876.894|srm.server.TSRMRequestCopyToRemote.upload|2|4
9|ab2bb06914ce590266bca5117c11ed91|1225524876.894|srm.server.req.to.status|2|4
10|d16c585253b2e420be8322fc64f759fc|1225524876.895|srm.server.tx.push|0|4
11|7b7370e35a8917ae5f976dc6f4f611f9|1225524876.895|srm.server.tx.push.size|2|4
12|dc1d0b07b1b404be2be3c2a7ddf23148|1225524935.733|srm.server.tx.push|1|4
13|ee8bc3e776910fae76f3ec310eaba879|1225524935.733|srm.server.req.to.status|2|4
14|d2c23970199473453820ac07aec5eeb7|1225524935.733|srm.server.req.to|1|4


You could also try out the R code in trunk/R/bestman.R in the NetLogger subversion repository. Just looking at the contained SQL should give you an idea how this data can be queried and joined together on those ‘guid’ attributes so you can look at all the information from one transfer together. Note that there are still some issues with getting the same GUID for all the events, and in particular the ‘queued’ event doesn’t seem to get the same GUID as the rest.

If you want to look at the source code for the parser itself, look under trunk/python/netlogger/parsers/modules/bestman.py and in particular the code section that mentions “version 2″. Hopefully it’s reasonably clear how events are being mapped and processed.

I found this page describing a way to do it with ESS (Emacs Speaks Statistics). So, I downloaded ESS from the ESS homepage and adapted the script slightly to be a simple shell-script.

#!/bin/sh
# Use emacs ESS pkg to indent R file
# Dan Gunter, Dec 2008
function usage () {
printf "Indent R file with Emacs ESS package.\n"
printf "Usage: $0 FILE\n"
exit 1
}
f=$1
shift
if test "x$f" = x -o "x$f" = "x-h"; then
usage
fi
emacs -batch \
-eval '(load "/Users/dang/local/lib/emacs/ess-5.3.8/lisp/ess-site")' \
-f R-mode \
-eval '(insert-file "'${f}'")' \
-eval '(set-visited-file-name "'"${f}"'")' \
-eval '(indent-region (point-min) (point-max) nil)' \
-f save-buffer \
2>/dev/null


R does not have the best interface to this, but I found online at a script to run R programs using #! syntax that uses the R startup sequence to “fool” the interpreter into running your script as part of its initial commands. I tweaked this a bit, to use printf instead of echo and to automatically quit the interpreter after the called script returns, and then checked it into subversion as “runR” in the trunk/R directory.

But how to take my current files of various functions and make them into a “real” executable? What I really need is the equivalent of Python’s optparse, or at least getopt. Time to do some more hunting..

# Start with a completely clean DB
mysql> drop database test;
mysql> create database test;

# Load in existing events from the existing log file (note the -C passed to nl_loader to create the schema)
$ grep " event_srm_log " pdsfgrid5.nersc.gov.vdt.log | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -C -u mysql://localhost -p db=test
# This took about 35 minutes to complete, yielding some 220K events in the DB

# Now the pipeline:
$ tail -f pdsfgrid5.nersc.gov.vdt.log | \
  grep " event_srm_log " | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -u mysql://localhost -p db=test

I’ve also done this for gram events for the tech-x guys…

+---------------------+
| gram_id |
+---------------------+
| 14094.1226615002 |
| NULL |
| 16255.1226616087 |
| 19674.1226619065 |
| 22903.1226620711 |
| 28114.1226625017 |
| 29889.1226625489 |
| NULL |
| 11191.1226683572 |
| 25930.1226693279 |
| 30603.1226697028 |
| 7016.1226699246 |
+-------------------- -+
12 rows in set (0.00 sec)