Project Title: Self-Configuring Network Monitor
Project Type: Base
PI: Deb Agarwal and Brian Tierney
__________________________________________________________________
Achieving the goals of high performance distributed data access and computing will require wringing the best possible performance from the networks. Most existing tools for end-to-end tests of network performance provide the end user little or no information from intermediate hops within the network. Without this information, the end-to-end system is unable to identify and diagnose problems within the network. The goal of this project is to design and implement a self-configuring monitoring system that uses special request packets to automatically activate monitors deployed at the Layer three ingress and egress routers of the ESNet network and within the end site networks. A principal design goal of the system is to provide components that are secure, easy to install, and easy to maintain so that the system does not add a burden to the network’s administration. This architecture will not require modifications to the application or network routing infrastructure. Archived monitoring data will help point the way beyond the handcrafted systems of network testbeds to a production environment that can routinely support high performance distributed applications. This passive monitoring system will integrate with active monitoring efforts and provide an essential component in a complete end-to-end network test and monitoring capability.
In this first year we have begun designing and implementing the self-configuring monitoring system. The monitors are being deployed at the Layer three ingress and egress routers of the ESnet network and within the end site networks. A principal design goal of the system is to provide components that are secure, easy to install, and easy to maintain so that the system does not add a burden to the network's administration. The software components that comprise the Self-Configuring Network Monitor include: a graphical user interface for requesting activation of the monitors; a library that handles activation packets; a monitoring daemon; a data collection and transmission mechanism; and a graphical user interface that displays the monitoring results. The first step was to specify the format of the special request packets that automatically activate monitoring along the network path between communicating endpoints.
In the first six months, we designed prototype versions of each of the components of the self-configuring network monitor. This prototype monitoring system provides a java interface for users to request activation of the monitor. Currently this interface allows only the source, destination, and port to be specified. The monitor packet is formatted and sent on the network in the direction of the destination traffic. The monitor daemon accepts tcpdump-like filter requests. The daemon watches for an activation packet. Upon receipt of an activation packet, it interprets the packet and then activates a filter for the designated data.
In this version of the daemon we can monitor both directions of a stream and all the traffic of a set of parallel connections such as would be generated by parallel ftp or parallel iperf. The headers from the monitored data stream are returned using a TCP connection to the source of the activation packet. The prototype monitor software also contains a graphical interface that provides a rudimentary display of the packet sizes and timestamps.
One of the core difficulties in this kind of monitoring system is making sure that the monitor is able to capture every packet that it is filtering for and not interfere on the network. To achieve the minimal interference goal, the monitor is designed to operate off a network tap. To achieve the goal of not losing any of the packets, we have made improvements to the underlying network interface driver to reduce interrupt overhead. The network interface is normally configured to interrupt for every packet but this causes packet loss. We have modified the network driver to coalesce the interrupts based on a timeout. This then brings the loss to zero. In addition, we worked on code that provides accurate timestamps for the packets when they arrive at the monitor. This timestamping requires the driver on the network card to save an indication of when the packet arrived so that when a set of packets is handed up to the kernel, an accurate timestamp for each packet can be calculated. This code has been implemented and we are testing and debugging it.
We have purchased the first four self-configuring network monitor boxes. The machines themselves were ordered with features that match the BRO and NIMI machines. We completed initial checkout of the machines and installed one on the LBNL DMZ, one on the NERSC DMZ, and one in the LBNL booth at SC2001. We used these machines to test our prototype software. We have since also installed at ORNL. We hope to make the next installation at either SLAC, ANL, or StartTap but are awaiting further testing and development of the software before deployment.
We have worked with Tom Dunigan of ORNL to install and test the SCNM box at ORNL, and have begun looking at data from the LBNL to ORNL network path. We have also started working with Tom Lehman from ISI, and will help him to install SCNM boxes at ISI East (Arlington, VA) and ISI West (Marina Del Ray, CA).
We are currently developing visualization tools for SCNM data. The initial tool is a real-time interface that allows the data to be viewed as it arrives. We are also working on a tool that provides tcptrace (www.tcptrace.org) plots for the monitor results and allows overlay of multiple results from the monitors on the path.
We have begun documenting and distributing all of the SCNM components (see: http://www-itg.lbl.gov/Net-Mon/SCNM_hw_sw.html)
We are also currently working on upgrades to the capabilities of the monitor. We recently added the ability to handle variable length TCP headers, and we are working closely with the developer of pcapd to find and fix several bugs. The ability to capture variable length headers allows us to capture complete headers including the option fields. This provides a means of capturing the options without risking the capture of data from the packet. We are also implementing additional activation options to allow further specification of the traffic to capture.
During the next six months, we will be concentrating on performing data analyses of SCNM trace data. We will continue to develop tools to visualize the data and will work to understand what can be learned by looking at this data, and compare this data with other sources of TCP performance data such as web100.
The other focus for the next several months is performance of the monitor itself. Currently pcapd is dropping data from high bandwidth streams. We will be working to identify the cause of these drops and to make pcapd more efficient by eliminating unneeded memory copies.
We also hope to begin to use the monitor to study various end-to-end network measurement techniques such as NCS and to verify their behavior in the interior of the network.
We are working closely with the Net100 project and the LBNL portion of the Bandwidth Estimation project to coordinate opportunities to monitor their active measurement traffic using the self-configuring network monitors. We are also working with NERSC, ORNL, and ESNet personnel to identify issues and deploy monitors.