We propose to extend research activities into Internet performance by expanding the reach and functionality of the National Internet Measurement Infrastructure (NIMI) probes. The project will add new and extend existing measurement capabilities for NIMI to improve its ability to capture multicast traffic, measure end-to-end and hop-by-hop bandwidths, provide more details on inter-packet delay variation, out-of-order and duplicate packets. We will leverage and extend the tools developed for the PingER and Beacon projects to enhance NIMIs analysis and reporting capabilities, in particular providing the ability for the user to select measurement type, time window, metric affiliation groups, and to drill down to more detailed tabular and graphical information. We will provide selected data to the Network Weather Service to assist with predicting future performance. Finally, we also plan to extend the measurements to include IPv6 network parameters, and to evaluate the impacts of Quality of Service (QoS).

Using first the existing and then the new capabilities of NIMI, we will measure both existing paths between ESnet sites and new and existing paths between ESnet sites and ESnet collaborator sites with high performance network links. This will include extremely high performance links such as those provided by the National Transparent Optical Network (NTON) and transatlantic/transpacific OC-3 to OC-12 links.

This active monitoring system will integrate with passive monitoring efforts and provide an essential component in a complete end-to-end network test and monitoring capability. It will also provide a platform for adding new active measurements from other proposed SciDAC projects.

3 Background and Significance

3.1 Importance & Relevance

The extraordinary network challenges presented by scientists, researchers and in particular high energy nuclear and particle physics (HENP) experiments has created a critical need for network monitoring to understand present performance, set expectations, trouble shoot, and to allocate resources to optimize/improve performance. An infrastructure to provide advanced IP based data transport services is largely in place in the N. American, W. European and Japanese research and education communities. However, there currently does not exist a well defined, always on, systematic, ubiquitous and automated approach to characterizing the quality of services parameters of all the components involved in data transport services from a source to a destination. See for example the recent special editorial on network traffc measurements and experiments [20] for a fairly comprehensive account [Chen00]. Current measurement infrastructure is not able to keep up with the increasing size of, demand for and reliance on the Internet. This problem only gets worse as more demanding applications are developed and deployed.

The growth of high performance, data intensive science applications has dramatically increased the need for reliable and regular network measurement. These applications, including those of HENP experiments such as BaBar [babar] and Atlas [atlas], and the Particle Physics Data Grid (PPDG) [ppdg], depend on the network measurement infrastructure to: trouble shoot the network; set realistic expectations; plan for experiments; and, understand overall performance such as throughput. These requirements include high performance bulk throughput for Terabytes of physics data, data replication, remote backup and archiving, and content distribution such as video streaming. In addition new services such as QoS, and new applications such as interactive voice over IP (VoIP), experiment control, multimedia and multicast applications are increasing the need for new types of measurements including the effects of applying QoS and measuring metrics such as jitter, continuous availability/reachability and multicast. For example: merely detecting reachability loss for multicast services is inherently a 2-dimensional, full-mesh problem.

3.2 Gaps to be filled by current proposal

The current proposal will bring together and leverage four existing and successful measurement projects, specifically PingER, NIMI, Beacon and NWS to create a measurement infrastructure which will support the growing demands of high performance data intensive science applications. The proposal also extends NIMI to make measurements of the new IPv6 and QoS enabled networks, and will help validate the scaling of the measurement techniques by extending the measurements to extreme high performance networks such as NTON. The proposal also has close ties to several other separately funded SciDAC proposals that will provide mutual benefit and synergy.

3.2.1 PingER

We will extend the highly successful PingER project, in particular by providing more detailed (more frequent as well as more metrics) information between critical sites with high performance network connections, and by an increased focus on high performance network ESnet sites and sites with strong ESnet site collaborative requirements. It will also provide an alternative measurement technique for paths where ICMP may be restricted [rate] and/or a way to validate whether ICMP rate limiting is being used on a path. At the same time it will leverage the analysis and reporting facilities of PingER.

3.2.2 NIMI

· The proposal also builds on the NIMI architecture that is now successfully deployed at about 51 sites (including three of the proposer sites: SLAC, PSC & LBNL) in 10 countries. We plan to extend two critical components of the NIMI architecture: first, to add additional resource control mechanisms and second to expand the reporting and analysis capabilities. For the reporting and analysis component, we will leverage the tools in use in PingER to provide public web access to tabular and graphical historical data, with selection of metrics, time scales, path groupings, and drill down to more details and graphs.

Results will be available publicly via the web. Example uses include:

· Enable network and applications users to have realistic performance expectations for existing and new applications;

· Provide trouble shooting assistance by identifying when changes occurred and what the changes were and in some cases what the impact of the changes may have been;

· Assist in verification that expected levels of service are being met;

· Provide input to setting and verifying service level agreements (SLA’s);

· Help decide which path to use when more than one is available, and;

· Assist in deciding where to locate a remote computing/replication facility. Some success stories of using PingER for the above purposes can be found in [examples].

3.2.3 Beacon

The IP Multicast Beacon [beacon] is a real-time monitoring system for IP Multicast reachability, loss, jitter, and time synchronization. While a real-time monitoring system is incredibly useful for debugging reachability problems, we really need a system to archive the IP Multicast state over time to watch how the service evolves over the long term. Some failures of the network are immediately evident from real time measurements, but other failures require longer term monitoring of IP Multicast traffic.

The IP Multicast Beacon [beacon] software has provided visibility into the state of deployment of IP multicast across several National backbones (Abilene, ESNet, vBNS) and several dozen local sites (National Laboratories, Universities, and other organizations). Adding Beacon functionality to NIMI probes will reduce the number of network probes necessary at a given site, while increasing the number of instrumented IP Multicast endpoints.

We propose to include IP Multicast Beacon functionality to the AIME framework. Each IP Multicast-enabled NIMI node will both transmit and receive IP multicast test traffic, as the current IP Multicast Beacon implementation [beacon] does today. The reachability information gained by these measurements will be archived using the same mechanisms as other NIMI measurements, and made available through the same data dissemination mechanisms (primarily web pages).

3.2.4 NWS

In addition the integration with the Network Weather Service will provide a base for ESnet application developers to instrument and improve their applications to take advantage of dynamic forecasts of performance characteristics. A key unfunded member of the current proposal’s team (Rich Wolski) is also the chief architect of the Network Weather Service.

The close contact of key people in this proposal with PPDG application developers will assist in extending mature applications such as parallel FTP to take advantage of the Network Weather Service (NWS) [nws]. In addition, we anticipate that the integration of NWS and NIMI capabilities will generate new network performance analysis and forecasting research results. This proposal will enable PPDG developers to leverage those results immediately.

3.2.5 IPv6

Another valuable contribution will be IPv6 monitoring. As IPv6 is deployed it will be useful to monitor the performance and peering arrangements. It is proposed that porting NIMI to IPv6 will be investigated and a small number of IPv6 aware NIMI boxes will be deployed. One of the key collaborators, Warren Matthews, is also a leader in monitoring IPv6 paths [ipv6meas] and SLAC is connected to the ESnet IPv6 testbed.

3.2.6 QoS

We will enable the NIMI measurement tools to mark packets and make measurements over paths that support QoS. The following two SLAC connections will assist in moving this forward: SLAC is connected to the ESnet QoS testbed; SLAC also has a joint (SLAC’s end is not funded) proposal [daresbury] with Daresbury Lab in England to investigate the effectiveness of QoS techniques, which will provide access to a transatlantic QoS controlled bottleneck.

3.2.7 NTON

Instrumenting the NTON with a few NIMI probes will enable us to understand whether and how the probes, the measurements and analysis can be scaled to an extremely high performance (OC48) network, and also assist in providing a better understanding of the end-to-end performance of the NTON. SLAC is an NTON site with an OC48 and has demonstrated > 900Mbits/sec throughput from Dallas to SLAC [sc2k] in November 2000. Also several major SLAC/Babar collaborators are connected to NTON, so there is significant interest in ensuring it works well for their major applications.

3.2.8 Ties with other SciDAC proposals

We will work closely with the LBNL passive measurement proposal team to understand how the active and passive sets of information complement one another and also mutually validate the measurements. In particular, we see considerable advantages in being able to request passive measurements while making some special active measurements. The passive monitoring capability provides a means of observing the traffic as it flows through the network and the active monitoring capability provides the means of generating and controlling that traffic and measuring the end-to-end result obtained. For more on comparing passive and active measurements and the complementarity of the two mechanisms, see Passive vs. active monitoring [passive].

We will also work closely with other network researchers to follow their developments and to assist with evaluation and integration of their tools into NIMI. Four examples of this, hopefully to be funded separately, are:

· To add tools and analysis from CAIDA-led Accurate Estimation of End-to-End and Hop-by-Hop Bandwidth Characteristics for measuring end-to-end and hop-by-hop bandwidth characteristics in the face of cross traffic [caida];

· To integrate the Rice University-led INCITE: Edge-based Traffic Processing and Service Inference fro high-Performance networks [incite] multi-fractal measurement tools into NIMI and make the data available to the researchers;

· To utilize the dynamic statistical experiment design tools proposed by ORNL, LANL and SLAC in Statistical Analysis and Design Methods for End-to-End Guarantees in Computer Networks to optimize the frequency and pairs being used in the NIMI measurements.

· To compare and contrast NIMI throughput measurements with simulation predictions such as those proposed in the SLAC-led Optimizing Performance of Throughput by Simulation (OPTS) project.

3.2.9 Relevance of current project

At an upper level, with the infrastructure in use, we expect to be able to assist in the following goals:

·Be able to understand/predict whether and how the application should work from a network standpoint

·Enable ESnet and ESnet collaborator network engineers to make promises that are related to user expectations (today’s promises are mainly best effort)

·If the application should work and it does not, then how does one make it work.

· If it shouldn't work then what has to be done

· Provide an understanding of how to build the network infrastructure needed to support a given application

·Make it increasingly likely that an application that should work, does work

· Reduce the time and effort to fix things:

· Make sure there are measurements available in the right places

Provide reports that produce information that can be passed onto to another discipline (- e.g. from user to site network engineer to ISP NOC), both for users and network engineers.

4 Preliminary Studies

4.1 Other related measurement projects

There are several existing Internet active end-to-end measurement projects in place today, including AMP [amp], NIMI [nimi], PingER [MC00], RIPE [ripe], skitter [skitter] Surveyor [surveyor], Beacon [beacon] and the NWS. A comparison of some of the projects [cf] indicates that most restrict themselves to measuring delays (or round trip times (RTT)), losses and routes. NIMI and the NWS on the other hand are mainly envisioned as infrastructures that enable monitoring of both delays and throughput. In addition the NWS is capable of generating real-time forecasts of future performance levels.

4.1.1 PingER

The existing DoE/MICS sponsored PingER project provides historical RTT, loss, and reachability information for over 3000 pairs of hosts in over 70 countries with data going back more than 5 years. These countries, in year 2000, had over 78% of the world's population and about 99% of the online users of the Internet The PingER project automatically gathers, archives, reduces the raw data into forms suitable for analysis, analyses the data and produces tabular and graphical reports and web pages. The reports are accessible world wide via WWW forms that will allow users to dynamically customize what information they wish to see, and more statically (for reports that cannot be generated sufficiently quickly to be interactive) by browsing the project's WWW pages. The current proposal should be regarded as complementary to the PingER project. Whereas PingER focuses on light-weight (low network impact) measurements covering a wide range of countries and networks, this proposal focuses on ESnet [esnet] and ESnet collaborator sites with high performance connectivity (i.e. mainly research and education sites in N. America, W. Europe and Japan with link speeds today of at least 10Mbps). For such sites more intensive monitoring is possible (due to the high performance links available) and needed (to characterize today’s high performance applications), than is provided by the low impact PingER monitoring. By more intensive we imply measurements that generate more traffic, such as making measurements more frequently, or performing high throughput measurements.

4.1.2 NIMI

NIMI (National Internet Measurement Infrastructure) is a software system for building network measurement infrastructures. The project is a joint collaboration with Berkeley Lab, and is currently funded by the Defense Advanced Research Projects Agency (DARPA) award #A0G205.~~DARPA~~. A NIMI infrastructure has two main components: a set of dedicated measurement servers running on hosts in a network and measurement configuration and control software running on separate hosts. Key NIMI design goals are: scalable to potentially thousands of NIMI servers; works across administrative and trust boundaries; accommodate and enforces diverse policies governing measurement access; and supports easy delegation of partial measurement access, to encourage different domains to participate in public uses of the infrastructure. NIMI servers queue requests for measurement for some future point in time, execute the measurement when its scheduled time arrives, store the results for retrieval by remote measurement clients, and delete the results when told to do so. NIMI does NOT presume a particular set of measurement tools. Instead, NIMI servers have the notion of a measurement module, which can reflect a number of different measurement tools. Security is a central architectural concern: all access is via public key credentials. The owner of a server can determine who has what type of access by controlling to whom they give particular credentials.

The existing NIMI project has successfully developed and deployed a pilot infrastructure to about 51 sites, mainly universities in the US, plus some universities and institutions in W. Europe and Korea. Today, very few (2) of these sites are directly on ESnet. NIMI is based on a collection of measurement probes that cooperatively measure the properties of Internet paths and clouds by exchanging test traffic amongst themselves. It provides: decentralized control of measurements; strong authentication and security; mechanisms for both maintaining tight administrative control over who can perform what measurements using which probes; delegation of some forms of measurements; and simple configuration and maintenance of probes. The NIMI probes can make ping, traceroute, throughput and muliticast measurements, and the platform is designed to make it simple to include other active measurement tools. Though the project has produced many important studies, currently there are no regularly updated NIMI measurement results available, on the Web for public access.

4.1.3 Beacon

RTPMON [rtpmon] was used to show IP Multicast reachability and loss rates during early deployment of Access Grid nodes [accessgrid]. Unfortunately, RTPMON assumes that the IP Multicast service of an internetwork is working correctly. The first attempt to overcome this limitation was to modify the IP multicast user applications to send their RTCP messages by IP Unicast to a central RTPMON receiver. This technique quickly indicated nodes not served by IP Multicast.

A suitably modified version of the vic [vic] software was quickly packaged up and sent to Access Grid sites that were experiencing IP Multicast deployment problems. Even in this rough state, the operational data provided in real time to network engineers was invaluable: for the first time, the network engineers could directly see the effect of changes across the whole set of participating nodes in an IP Multicast group.

Operationally, however, this initial capability was less than ideal for several reasons:

· The modified vic required that the user of the host computer remain logged

· The modified version of vic had very specific host requirements in terms of operating system and support programs.

· The modified RTPMON was useful only on the machine where the modified vic clients were programmed to send their reports.

NLANR [nlanr] agreed to implement a Java [java] based software package--known as the IP Multicast Beacon--to replace the modified vic and RTPMON setup. The “write once, run anywhere” features of Java eliminated the host dependencies that plagued the earlier RTPMON/vic approach. Each instrumented node requires only a Java runtime, which often comes bundled with the operating system. The central collection point exports its data in real time as a web-browser accessible page, no longer requiring special software for network engineers and managers to access the data.

The existing NLANR Beacon software provides real-time representation of IP Multicast reachability, loss, delay, and jitter. But it does not provide any history mechanism to view these variables from moment to moment.

4.2 Experience and Competence

4.2.1 SLAC

SLAC is the home site for the BaBar experiment/collaboration and is associated with several other projects requiring high network performance such as the SSRL/SDSC collaboration and the GLAST collaboration. ANL is a Globus core site and is also a major collaboration site for the LHC/Atlas project. Both ANL and SLAC are PPDG collaborators with one of the PIs for the PPDG being at SLAC, and the SLAC PI for the current proposal is currently a member of the PPDG. The close association with HENP user and developer communities requiring high network performance and with the PPDG and other Grid activities will provide a fertile ground for understanding needs and requirements, trying out ideas, and providing feedback.

SLAC is home site for the highly successful IEPM/PingER active Internet End-to-end Performance monitoring project, arguably the most extensive project of its kind in the world today. The leader of the IEPM/PingER project and its PI are a key member and a PI of the current proposal.

4.2.2 PSC

The Pittsburgh Supercomputing Center (PSC) is the lead institution on the NSF-sponsored Web100 project. The goal of Web100 is to drive the wide deployment of advanced TCP implementations that can support high performance applications without intervention from network experts. This is accomplished through extensive internal instrumentation of the stack itself plus TCP autotuning. This will permit simple tools to "autotune" TCP as well as diagnose other components of the overall

system such as inefficient applications and malfunctioning Internet paths. Web100 will bring better observability to these last two classes of problems, leading to long term overall improvements in both applications and the underlying infrastructure.

PSC has participated in the Engineering Services (ES) group of the NSF-sponsored NLANR project for over six years. Through NLANR ES, PSC provides ongoing engineering and technical support to NSF-funded High-Performance Connection (HPC) sites as well as the broader high-performance networking community. NLANR ES staff provide expert consulting for HPC campus network engineers, develop documentation on emerging HPN technologies, provide tools for diagnosing network performance, and provide a forum for disseminating information to the high-performance network community through the NLANR/I2 Joint Techs Workshops. The NLANR ES performance tuning WWW page http://www.psc.edu/networking/perf_tune.html is an authoritative source

for information on how to hand-tune a wide variety of operating systems for use over the national high-performance network backbones.

PSC has made a number of contributions to TCP technology. They were the first authors on RFC2018, "TCP Selective Acknowledgement Options", which enable the TCP sender to know exactly which data is missing at the receiver. Their ongoing research on TCP congestion control has resulted in the development of several algorithms that can improve TCP performance in high-performance environments [MM96, MSJ99]. They also developed the base model for TCP bulk

performance [MSMO97], have made substantial contributions to the IETF IP Performance Metrics working group [Ma96, RFC2330, MA99, and Mat99] and did groundbreaking work in TCP Autotuning [SMM98].

4.2.3 ANL

ANL has been driving deployment of IP Multicast in the networking community as a side effect of the Access Grid [accessgrid] deployment effort. We have successfully deployed IP Multicast as part of the basic infrastructure at major networking conferences such as iGrid and SC, where Access Grid installations were demonstrated and used. We have helped push IP Multicast into institutions ranging from National Laboratories to Native American tribal colleges.

Our primary diagnostic tool for this IP Multicast deployment work has been the IP Multicast Beacon [beacon], and based on our experience we believe in the acute need for an archive of Beacon state that can be queried over time. Getting IP Multicast deployed on a “hero” basis is one thing, but ensuring it stays up and stable over time requires even more sophisticated tools than we have today.

5 Research Design and Methods

Our proposal has five main research goals:

a. the enhancement of NIMI monitoring/measurement capabilities to include new, or extend existing measurements of vital network performance characteristics such as end-to-end and hop-by-hop bandwidth estimation, inter-packet delay variation, duplicate and out of order packet delivery frequencies, cross-traffic estimation, and multicast performance;

b. research into and development of tools to provide automated analysis, and reporting tools with the generation of web accessible pages providing the user with interactive access to results for selected metrics, measurement methods, time-scales, affinity aggregation groups, and with drill down to graphical plots and more detailed results;

c. expansion of NIMI resource control capabilities, including research into how to allocate system resouces in order to support multiple measurement studies without adversely effecting measurement results.

d. the provision of forecasting capabilities within the monitoring framework, and

e. the deployment of NIMI probes to selected high performance sites of critical interest to the ESnet community.

The resulting tools and NIMI probes will serve both as an invaluable resource enabling the development of new HENP applications and as a network research tool of unmatched scope and capability.

5.1 Extension of NIMI Monitoring Capabilities

WeE will begin the project using the existing NIMI tools to measure: unicast delay, loss, and jitter; multicast routes; traceroutes; TCP throughput and bulk transfer capacity. The NIMI probes will also contain a packet filter that will enable looking at the probe’s traffic only (for privacy/security reasons). We will research and extend these tools to measure inter packet delay variability, and reordering. We will also investigate pathologies such as duplicate packets, conditional loss probabilities (i.e. the probability that if one packet is lost the next packet is also lost) and the distributions of numbers of adjacent packets lost (the conditional loss probabilities and gap metrics defined in [bolot]). We plan to research and add new tools to NIMI for measuring bottleneck bandwidth, cross-traffic, and to examine dynamically optimizing the measurement frequencies and which sites are monitored using statistical experiment design techniques. We also plan to integrate the Beacon functionality into the NIMI framework. In addition to simple integration, we will integrate the data generated by the Beacon system into the NIMI long-term database. We will extend the deployment and measurements to include some IPv6 [ipv6] paths and paths that support QoS. We will also work with ESnet folks to investigate the use of NIMI to look at link utilization at site border routers by accessing the SNMP MIBs in read only mode. On-demand measurements of bulk throughput capability will also be added to help understand and trouble shoot bulk throughput applications, and to validate more simple/lightweight bulk-throughput estimators (e.g. simulation). Other measurements such as HTTP, may be added depending on demand.

5.2 Automated analysis and reporting

Currently, measurement results produced via the NIMI measurement infrastructrue are packaged and shipped to a predetermined repository referred to as the Data Analysis Client (DAC). Post-processing of the results, when performed, are done outside of the NIMI architecture. In order to generate real-time feedback for the user, we propose to implement automated post-processing on the DAC. This work will include first research on and then development of analysis profiles to control the post-processing of an individual measurement orf subsets of a measurement study. Moreover, we will research, test and develop estimators for the data, and to summarize and visualize the estimators and produce web accessible reports that will include tabular time series similar to those provided by PingER [report]. The reports will allow user selection of metric (e.g. delay, loss, jitter, throughput, reachability), time scales (both the time separation of the adjacent points, and the window in time being reported on), paths (both the source and destination and grouping by affinities such as collaboration, geographical region, Internet Service Provider (ISP)). The user will also be able to sort the data by simply clicking on column headings. In addition the reports will provide user drill down to display time series plots and frequency histogram details for individual paths and groups of paths. The tabular data will also be exportable to applications such as Excel to enable customized analysis and reports to be generated by interested users. In addition traceroute history information will be summarized and made available. We will further study and research the results to see how the estimators scale to ultra high speed networks, understand how to visualize the more frequent measurements (compared to PingER), research, develop and test computationally efficient new estimators, and compare and contrast the NIMI results with simpler, more lightweight mechanisms. The raw traceroute data will also be made available to assist with research efforts elsewhere, such as the network tomography research proposed as part of the INCITE: Edge-based Processing and Service Inference for High-Performance Networks project proposal.

5.3 NIMI Resource Control

Although the current NIMI infrastructure can support multiple measurement studies, researchers who initiate the measurements must coordinate their efforts in order to avoid biases produced by concurrently running measurement tools. Moreover, the NIMI probes themselves can become unstable if aggressive or poorly architected measurement tools consume system resources. To mitigate the need for researcher pre-communication, and to make the NIMI probes more robust, we intend to research, develop and implement several architectural improvements in resource control and enforcement, including:

· Fine-grained access control, to limit the resources (i.e., packets on the wire, disk space usage) on a per-NIMI probe, per-user basis.

· Inter-domain monitoring via the NIMI Configuration~~entral~~ Point of Contact~~rol~~ (CPOC), where the CPOC periodically polls all NIMI probes within its administrative domain for system resources. This will act as an early warning system for the human administrator.

· Measurement tool resource profiles (e.g., ports, protocol, Berkeley Packet Filters (BPFs)), to allow the scheduler to check for competing resources and compensate accordingly.

· Generic packet filter, being developed at LBNL, to act as an enforcement proxy between the measurement tools and the BPF devices.

· Measurement expirations, including a fine-grained notion (i.e., how long to attempt to deliver a measurement, how long to keep a result around on local disk), as well as more granular expirations (i.e., over the entire measurement study per NIMI probe).

5.4 Forecasting

We plan to provide selected data to the NWS to assist with predicting future performance. While network monitoring alone is critical to network capacity-planning and diagnostic activities, if HENP applications are to use the resulting data for scheduling, a forecast of future performance levels is required. The most sophisticated network performance analysis techniques available today indicate that the observed performance of critical network links can change rapidly. If application scheduling decisions are based on recently-observed performance conditions the scheduler must assume that the conditions will persist until the application executes. If conditions change between the time that the schedule is developed and the application executes, the schedule will be based on assumptions of resource availability that are no longer true. That is, the schedule is developed for performance that was, but no longer is, available when the application eventually executes. To support application scheduling, then, the performance system must be able to develop predictions of future performance levels. As such, we plan to provide an interface between NIMI performance monitoring facilities and the NWS. We will use this interface both to study the problem of real-time throughput forecasting from network-level performance measurements, and to support HENP application scheduling.

5.5 Deployment

We propose to extend the deployment of the existing NIMI infrastructure, ~~(The NIMI project is based on work sponsored by the Defense Advanced Research Projects Agency award #AOG205)~~ originally funded by ~~Defense Advanced Research Projects Agency (~~DARPA) and the National Laboratory for Applied Network Research (NLANR), to enable gathering of data from sites of critical interest to ESnet, HENP, and this proposal. We will strategically deploy the NIMI probes to optimize their usefulness at measuring unicast and multicast traffic (i.e, outside of firewalls). Initially, additional probes will be administered by the existing NIMI team. Administrative support consists of; installing and initially configuring the software, monitoring the system, upgrading and adding measurement tools, and delegating researchers access to all or parts of the system. Eventually, as the architecture improves (mainly as outlined in section 5.3), the system should be robust enough to partition the ESnet NIMI infrastructure into one or more logical administration domains where each domain would then be responsible for administering the NIMI probes contained within. The creation of a separate ESnet domain is in keeping with NIMI's fundamental design goal to support administratively heterogeneous infrastructures.

We believe that there are several projects that would like to deploy network measurement boxes at critical points in the network thus we propose to deploy boxes (hardware, operating system and network connectivity) that can be used by the other projects. Other measurement proposals would be able to make use of the probe hardware, operating system and network interface to perform their measurements.

We also plan to extend the current project to instrument the National Transparent Optical network (NTON) [nton] testbed with our tools to ensure the tools and estimators scale to the next generation high speed networks and to assist with understanding the NTON performance.

The data from the NIMI probes will be automatically gathered by a central archive machine that will retrieve and store the measurements in a file system. The archive machine can also host some default data analysis client/applications, however in the interests of scalability and simplicity we plan to separate this task out to other hosts. Reports from the applications will be made available via the web so there will also be a web site host with powerful tools to enable a user to navigate to find the information of interest.

5.6 Tentative timetable

5.6.1 First 6 Months

5.6.1.1 SLAC

The first phase (roughly months 1 to 6) of the work for SLAC will be, with some assistance from Vern Paxson and PSC, to study and understand NIMI in detail. In particular this will include the existing measurement tools, how they work (scheduling and how they probe the network), what they record and how (formats, file hierarchies, databases). We will also review the existing data extraction, analysis and reporting tools. To assist in this, we plan that there will be some face-to-face meetings between the implementers, to facilitate getting the project started and initiate good relationships between the people involved. As we gain an understanding of the recorded data we will start to extend the analyses and reporting. Given our experience with ping this may start with ping by adding conditional loss probability, packet length gap distributions, reordering and duplicate packet reporting. We will extend the PingER analysis tools and infrastructure to incorporate existing NIMI ping measurements inter-packet delay and the above additions. At the same time we will be working with two or three other friendly sites (including ANL and the University of Tennessee at Knoxville) to agree upon and arrange for early deployment of NIMI probes at their sites.

5.6.1.2 PSC

PSC will start to do research on how improve the remote manageability/control of the NIMI measurements.

5.6.1.3 ANL

The ANL team will start to look at how to utilize the existing NIMI multicast measurements and to evaluate how to add Beacon functionality. The University of Tennessee team will develop an initial interface between the NIMI performance gathering infrastructure and the NWS forecasting subsystem.

5.6.2 Next 6 Months

5.6.2.1 SLAC

During the next phase (roughly months 7 to 12), the SLAC team will extend the reporting of NIMI data to include the throughput and bandwidth measurements. During this phase we will investigate various existing tools (e.g. [traceping], [surveyor]) for analyzing and reporting on the NIMI traceroute information we have gathered, choose one and use it to make our traceroute information available. SLAC will also work, in some cases with other proposals, to start to look at how to extend the NIMI measurements, for example by adding a hop-by-hop or a bottle-neck or end-to-end bandwidth estimation capability such as pchar [pchar], pipechar [pipechar], nettimer [nettimer] and / or working with CAIDA to add hop-by-hop and end-to-end bandwidth measurements [pathrate].

5.6.2.2 PSC

PSC will extend the NIMI probes to SciDAC High Performance network measurement/analysis sites such as ORNL, Rice, SDSC, Caltech, UTK, FNAL, BNL, Jlab, SDSC and Wisconsin. They will also start to deploy and test the new remote management tools to the NIMIs.

5.6.2.3 ANL

The ANL team will start to integrate Beacon into its NIMI and provide analysis and reporting. The University of Tennessee team will deploy an enhanced NWS capable of serving NIMI-gathered data via the NWS, and will begin the study of new forecasting capabilities.

5.6.3 2^nd year

5.6.3.1 SLAC

In the second year, SLAC will add IPv6 measurements, analysis and reporting to the NIMI suites. We will also work with LBNL to understand how to tie together the passive and active measurements, e.g. how to co-schedule them, how to compare, contrast and co-validate the measurements. In addition we will study the extra information available from the joint measurements and learn what they are and can tell us, and study how to get the most value from them.

5.6.3.2 PSC

PSC will complete the deployment on NIMIs started earlier to the remaining SciDAC sites. We will include some IPv6, and non US sites (e.g. IN2P3 in Lyon France, a UK Lab such as RAL or DL, KEK in Japan, and an INFN site in Italy) in this deployment, the latter to assist in transatlantic / transpacific measurements.

5.6.3.3 ANL

ANL will also extend the measurements/analysis/reporting suites to add multicast measurements. During this period we will evaluate if and how to integrate our measurements with those from Surveyor or other measurement projects.

5.6.4 3^rd Year

In the third year, we will carefully validate and compare the measurements versus other mechanisms, and add new measurement suites based on our experiences and the experiences of others. We will also investigate providing access to the measurements and analyses to trouble shooting tools to make problem isolation much easier for non-expert users. We will evaluate how to provide a smooth transition to a production service as opposed to a project, put together documentation of procedures, resources needed, come up with a plan for transition, and work with others (as identified) to help identify (e.g. assist in proposal writing) how to provide an ongoing production service.

5.7 Subcontract or Consortium Arrangements

This project is a joint effort between SLAC, PSC and ANL, with unfunded assistance from Rich Wolski of the University of Kentucky and Vern Paxson of ACIRI/LBNL.

PSC is the main development and administration site for the NIMI. PSC will procure, configure and deploy and manage the extra NIMI probes. This will ensure a common base (hardware and software) with the existing NIMIs, which will make future management simpler. Since the PSC is the repository of expertise on NIMI they will also provide assistance to other researchers and developers to assist in modifying existing NIMI tools and adding new ones. PSC will also research how to enhance the NIMI architecture in order to provide automated result analysis and more robust resource control (i.e. minimizing measurement bias, stabilizing the system).

Linda Winkler and Bill Nickless of ANL continue to push IP Multicast deployment as part of the Access Grid [accessgrid] project. ANL will integrate basic IP Multicast Beacon [beacon] functionality into the NIMI software, and will go on to put Beacon data into the NIMI data collection so that it can be accessed over time.

Rich Wolski of the University of Kentucky is the main architect of the Network Weather Service (NWS). The synergy between the current proposal and the NWS will enable this proposal to provide measurements to the NWS and the NWS to analyze and use its existing tools to provide real-time forecasts back to the HENP and ESnet developers.

6 Literature Cited

[accessgrid]: http://www.mcs.anl.gov/accessgrid

[amp]: http://amp.nlanr.net/AMP/

[atlas]: http://www1.cern.ch/Atlas/Welcome.html

[babar]:http://www.slac.stanford.edu/BFROOT/

[beacon]: http://dast.nlanr.net/projects/beacon

[bolot]: G. Bolot, Characterizing End-to-End Packet delay and Loss in the Internet, Journal of High-Speed Networks, vol 2, no. 3, pp. 305-323, Dec 1993.

[caida]: http://www.caida.org/~kc/Props01/doeBW.html

[cf]: http://www.slac.stanford.edu/comp/net/wan-mon/iepm-cf.html

[Chen00]: T. M. Chen. Network traffic measurement and experiments: Guest Editorial. IEEE Communications Magazine, page 120, May 2000.

[daresbury]: http://dlitd.dl.ac.uk/public/i2qos/

[esnet]: http://www.es.net/

[examples] http://www-iepm.slac.stanford.edu/pinger/uses.html

[ipv6]: http://www.ipv6.org/

[ipv6meas]: http://www.slac.stanford.edu/grp/scs/net/talk/ipv6-oct00/IPv6-2000_files/v3_document.htm

[java]: http://www.java.sun.com

[Ma96] M. Mathis, Diagnosing Internet Congestion with a Transport Layer Performance Tool, Proceedings of INET'96, June 1996, Montreal, Quebec, Canada.

[MA99] M. Mathis, M. Allman, Empirical Bulk Transfer Capacity, Internet-draft: draft-ietf-ippm-btc-framework-02.txt, Work in progress revised October 1999.

[Mat99] M. Mathis, TReno Bulk Transfer Capacity, Internet-draft: draft-ietf-ippm-treno-btc-03.txt, Work in progress revised February 1999.

[MC00]: W. Matthews & R. Cottrell, The PingER Project: Active Internet Performance Monitoring for the HENP Community, IEEE Communications, 38(5), May 2000.

[mcast]: A. Adams et. al. The Use of End-to-end Multicast Measurements for Characterizing Internal Network Behavior, IEEE Communications, 38(5), May 2000.

[MM96] M. Mathis, J. Mahdavi, TCP Rate-Halving with Bounding Parameters, Technical Note, October 1996.

[MSJ99] M. Mathis, J. Semke, J. Mahdavi, The Rate-Halving Algorithm for TCP Control, Internet Draft: draft-mathis-tcp-ratehalving-00.txt, August 1999, Currently in "Last Call" for experimental standard RFC status.

[MSMO97] M. Mathis, J. Semke, J. Mahdavi, T. Ott, The Macroscopic Behavior of the TCP Congestion Avoidance Algorithm, Computer Communication Review, volume 27, number 3, pp. 67-82, July 1997.

[nettimer]: http://mosquitonet.stanford.edu/~laik/projects/nettimer/

[nimi]: http://www.ncne.nlanr.net/nimi/

[nlanr]: http://www.nlanr.net/

[nton]: http://www.ntonc.org/

[nws]: http://nws.npaci.edu/NWS/

[pchar]: http://www.employees.org/~bmah/Software/pchar/

[passive]: http://www.slac.stanford.edu/comp/net/wan-mon/passive-vs-active.html

[pathrate]: http://www.caida.org/outreach/papers/consti.pdf

[pipechar]: http://www-didc.lbl.gov/pipechar/

[ppdg]: http://www.cacr.caltech.edu/ppdg/

[rate]: http://www-iepm.slac.stanford.edu/monitoring/limit/limiting.html

[report]: http://www.slac.stanford.edu/cgi-wrap/pingtable.pl?tick=monthly&from=PPDG&to=PPDG

[RFC2018] M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow, TCP Selective Acknowledgement Options, Internet Request for Comments 2018 (rfc2018.txt) October 1996.

[ripe]: http://www.ripe.net/ripencc/mem-services/ttm/index.html

[rtpmon]: http://bmrc.berkeley.edu/~drbacher/projects/mm96-demo/

[sc2k]: http://www-iepm.slac.stanford.edu/monitoring/bulk/sc2k.html

[skitter]: http://www.caida.org/tools/measurement/skitter/

[SMM98] J. Semke, J. Mahdavi, M. Mathis, Automatic TCP Buffer Tuning, ACM SIGCOMM '98 Computer Communication Review, volume 28, number 4, October 1998.

[surveyor]: http://www.advanced.org/csg-ippm/

[traceping]: http://av1.physics.ox.ac.uk/www/traceping_description.html

[vic]: http://www-mice.cs.ucl.ac.uk/multimedia/software/vic/

7 Budget

Insert the budget spreadsheets and justifications here.

8 Other Support of Investigators

8.1 SLAC

Les Cottrell expects support from this proposal - 10%. Other requested SciDAC support: Consortium for End-to-End Network Assurance Performance Research - 10%; Statistical Analysis and Design Methods for End-to-End Guarantees in Computer Networks – 10%; Edge-based Traffic processing and Service Inference from High-Performance Networks – 10%; Optimizing Performance for Throughput by Simulation – 10%.

Les Cottrell currently receives support from SLAC Computer Services - 80% and from IEPM/PingER project – 10%, and 10% from the PPDG project. The latter will go away the end of Fiscal year 2001.

Warren Matthews expects support from this proposal – 10%. Other requested SciDAC support: Consortium for End-to-End Network Assurance Performance Research - 10%; Statistical Analysis and Design Methods for End-to-End Guarantees in Computer Networks – 10%; Edge-based Traffic processing and Service Inference from High-Performance Networks – 10%; Optimizing Performance for Throughput by Simulation – 10%.

Warren Matthews currently receives support from SLAC Computer Services – 10% and from IEPM/PingER project – 90%.

1.1 PSC and ANL

8.2

Andrew Adams expects support from this proposal -50%. Andrew Adams currently receives support (50%) for NIMI on a subcontract to LBL (DAPRA funding), through September 2001. He also receives support (~~30%) for~~ ~~NIMI~~ ~~as part of the NSF Funded NLANR Engineering Services project as well as provides production networking support (30%) for the PSC managed Pittsburgh GigaPoP.~~

Gwendolyn Huntoon is not receiving any support for the project, but will provide general managerial and administrative oversight for PSC's component of the project as part of her role as Assistant Director at PSC.Insert attachments here

~~ANL~~

~~See attachment.~~

9 Biographical Sketches

9.1 Roger Leslie Anderton Cottrell

Stanford Linear Accelerator Center, Mail Stop 97, P.O. Box 4349, Stanford, California 94309
Telephone:	(650) 926 2523	Fax:	(650) 926 3329
E-Mail:	cottrell@stanford.edu

9.1.1 EMPLOYMENT SUMMARY

Period	Employer	Job Title		Activities
1982 on	Stanford Linear Accelerator Center	Assistant Director, SLAC Computing Services		Management of networking, telecommunications and computing
1980-82	Stanford Linear Accelerator Center	Manager, SLAC Computer Network		Management of all SLAC’s network computing activities
1979-80	IBM U.K. Labs., Hursley, England	Visiting Scientist		Graphics and intelligent distributed Workstations
1967-79	Stanford Linear Accelerator Center	Staff Physicist		Inelastic e-p scattering experiments, physics and computing
1972-72	CERN	Visiting Scientist, Staff Physicist		Split Field Magnet experiment
1972-73	CERN	Visiting Scientist		Split Field Magnet experiment
9.1.2 EDUCATION SUMMARY
Period	Institution		Examinations
1962-67	Manchester University		Ph.D	Interactions of Deuterons with Carbon Isotopes
1959-62	Manchester University		B.Sc. Hons.	Physics