xdrlib

Thinking about how to shuffle log events between Python and other languages (e.g. Java), XDR seems like one possible solution. It would allow non-string data types (i.e. encode the timestamps as floats) and minimize parsing on the receiver. So I fooled around a little with python’s xdrlib. Pretty nice, although it assumes an RPC model of one message at a time. I saw an interesting proposal to add a streaming read() function on the Python bug tracker, but that seems to me like overkill (especially since I intend to do my reading from Java).

Instead, I used a simple buffer length header. Here are the write_buf(), read_buf() functions. I’m using files here, but sockets should be the same (except for the usual issues with read() not returning as many bytes as you want).

def write_buf(ofile, p):
    """Write the contents of xdrlib.Packer, p,
    to ofile with a 4-byte header with the buffer size.
    """
    buf = p.get_buffer()
    buflen = len(buf)
    p.reset()
    p.pack_int(buflen)
    ofile.write(p.get_buffer())
    ofile.write(buf)
    return buflen

def read_buf(ifile, p):
    """Read a buffer from ifile into the xdrlib.Unpacker, p,
    using the 4-byte header to determine the data buffer size.
    """
    hdr = ifile.read(4)
    if not hdr:
        return 0
    p.reset(hdr)
    bufsize = p.unpack_int()
    data = ifile.read(bufsize)
    p.reset(data)
    return bufsize