Poor man’s pipeline

As a short-cut to getting a full nl_pipeline running (because I suck) here’s what I just did to load in from scratch all the SRM logs and start a “poor man’s pipeline” to pick up events as they come in:

# Start with a completely clean DB
mysql> drop database test;
mysql> create database test;

# Load in existing events from the existing log file (note the -C passed to nl_loader to create the schema)
$ grep " event_srm_log " pdsfgrid5.nersc.gov.vdt.log | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -C -u mysql://localhost -p db=test
# This took about 35 minutes to complete, yielding some 220K events in the DB

# Now the pipeline:
$ tail -f pdsfgrid5.nersc.gov.vdt.log | \
  grep " event_srm_log " | \
  nl_parser -m bestman -e '\S+ \S+ \S+ ' -m bestman -p version=2 | \
  nl_loader -u mysql://localhost -p db=test

I’ve also done this for gram events for the tech-x guys…