Internet Monitoring in the HEP Community. ^{∗}

Warren Matthews, Les Cottrell

Stanford Linear Accelerator Center, Stanford, CA 94309, USA.

David Martin

HEP Network Resource Center, Fermi National Accelerator Laboratory, Batavia, IL 60510, USA.

September 8, 1998

Abstract

This paper details the current work involved in providing the HEP community with an accurate measurement of the end-to-end performance individuals may expect by monitoring the performance of given Internet links. The HEP Internet Monitoring Project (known as PingER) [?] attempts to measure the performance of the Internet used by the High Energy Physics Research community by monitoring the packet loss and round trip time (RTT) of ICMP Ping packets from 17 monitoring sites around the world to 373 nodes at 267 sites across 1084 links involving 27 countries.

1 Introduction

The main work involves sending ICMP ping packets from 17 Monitoring sites in 8 countries across a total of 1084 links in 27 countries, and logging the results for further analysis. The PingER project is probably the largest monitoring project in existence. There are 373 remote nodes at 267 sites being monitored. 7 Monitoring sites are in the United States. The Stanford Linear Accelerator Center (SLAC), the HEP Network Resource Center (HEPNRC) at Fermi Lab, the Department of Energy (DOE) in Washington, Brookhaven National Laboratory (BNL), Atmospheric Radiation Measurement (ARM), Carnegie Mellon University (CMU) and University of Maryland (UMD). There are 2 further sites in North America in Canada at the TRIUMF facility near Vancouver and at Carleton University in Ottowa. There are also 6 sites in Europe, at the European Center for Particle Physics (CERN), the Deutsches Elektronen Synchrotron (DESY), the Rutherford Appleton Laboratory (RAL) near Oxford in England, the Niels Bohr Institute (NBI) in Denmark, CNAF and KFKI. Finally, there are 2 monitoring sites in Asia at Sinica in Taiwan and the KEK facility in Japan. In addition, the monitoring sites and remote nodes are connected via a large number of networks. In the United States, the Department of Energy funded laboratories (SLAC, HEPNRC, DOE, BNL, ARM) are connected by the Energy Sciences network (ESnet), and many US Universities (CMU) are connected to the very-fast Backbone Network Service (vBNS). In Canada TRIUMF is connected to BC-Net and Carleton University is connected via ON-Net, BC-Net and ON-Net are connected to the Canadian Canarie backbone. Networking in Europe is characterized by National Research Networks (NRNs) such as the Joint Academic Network (JANet) in the UK, frequently interconnected with the TEN-34 (soon to be upgraded to TEN-155) backbone.

1.1 HEP and Internetworking

The huge growth and commercialization of the Internet in recent years, means academic and HEP traffic must traverse the same congested paths and queue at the same congested routers as all other traffic. Monitoring our links provides vital information on the state of our connectivity and the feasibility of the HEP community’s future plans for networking.

∗Presented at Computing in High Energy Physics, August 1998.

1.2 Beacon Sites

Previously, all monitoring sites have been free to ping sites of interest to themselves. Recently however, the convept of beacon sites has been introduced, and all monitoring sites have been requested to ping them. These 50 beacon sites are considered to be of importance to HEP in general, and monitoring them from all 17 monitoring sites gives better information for trouble shooting and understanding the network in general.

1.3 Ping

Ping [?] is an ICMP (Internet Control Message Protocol) echo request, and it is part of the TCP/IP implementation and runs in the kernel. There is no need to install other applications and it is currently our best tool to indicate the true performance of the network, independent of the load on the machine that is running it.

1.4 Packet Loss

Studies show a small amount of packets are lost due to transmission errors, approximately 1 in 5000 packets arrive corrupted [?], but by far the main cause of packet loss is congestion in routers. Packets must queue to be processed, and if the queue is full, the packet is discarded. Polite protocols such as TCP back-off if this happens, and slow down the rate of transmission. If the originating host does not recieve an acknowledgment that the packet was received within 3 seconds, it is re-sent.

1.5 Round Trip Time

Ideally, traffic should traverse the Internet at the speed of light in glass (for fiber connections, slightly slower in copper). However, connections very rarely do. The major cause of delay in transmission is queuing at routers. The major effect of poor response time is on interactive sessions such as telnet where the session will hang, perhaps for quite some time.

2 Data Gathering

Each Monitoring Site pings the remote sites every half an hour and the results of each sample are written to a file. Each day the archive site at HEPNRC retrieves the data and stores it in a databasem

3 Analysis Method and Tools

Analysis is done on the data by each monitoring site to show short term reports in the form of a table with the packet loss and response time of the latest data. Furthermore each site provides the raw data and a simple configurable graphing tool. Also, a collective analysis is done on all the data at the analysis site at SLAC, providing detailed hourly, daily and monthly reports.

WWW CGI front end, graphs generated by SAS and perl, also use spreadsheet. show hourly trend over the course of the month, Monthly Report with Daily Ticks Summary Report with Monthly Ticks Group by site, tld, geographical location, userdefined groups

4 Results

Many links have improved this summer.

Further to the specific examples given here, it should be noted that the overall performance of the network improves dramatically during the main University holidays at Christmas, Easter and during the summer.

4.1 Performance in North America

Performance to sites on ESnet is good, with packet loss typically less than 0.5% and round trip times in the region of distance/0.6c. Sites on the vBNS Figure 1: The Holiday Effect: Performance between ESnet and Sites in the UK

Figure 2: Performance on Links between Sites in North America Figure 3: Performance on Links between sites in Europe

are also good, with packet loss typically less than 1%. ESnet peers with vBNS in many locations, traffic from SLAC to vBNS usually travels via Chicago. Performance to US Universities on neither ESnet nor vBNS and performance to sites in Canada are very variable. Connectivity to some sites, such as utk.edu (University of Kentucky) is good because it is connected to Oakridge National Laboratory which is an ESnet site and ESnet routing policy dictates traffic remains on ESnet for as long as possible, even if the ultimate destination is not on ESnet. Other sites are connected via commercial providers, and can suffer high packet loss and poor response times. The situation for all North American academic sites should improve dramatically over the next year or so, with the introduction of new high speed links as more Universities join vBNS and other Internet II projects.

4.2 Performance in Europe

Performance within most of West Europe is good, although packet loss to Spain is significantly higher than other parts, and the trend is increasing, although recent data to the Spanish beacon sites indicate the situation may be improving. Links to East Europe are frequently saturated, leading to large packet loss and poor response time.

Figure 4: Transoceanic Performance from Sites on ESnet

4.3 Performance in Asia

Performance inside Japan is good, but it is difficult to compare performance based in the geographic region of Asia because the connectivity is not geographically based. Traffic from KEK to Taiwan for example, travels via MCI’s San Francisco hub.

4.4 Transoceanic Performance seen from ESnet

Performance across the oceans is the usual bottleneck for traffic. Performance to the U.K. has always been very variable, inproving during University holidays and briefly when increased bandwidth is installed, but quickly returning to near saturation. The U.K. Research Network (JAnet) has recently increased bandwidth again, now to 90Mb/s. Overall the trend is 1-2% improvement per month. Performance to the rest of west europe (Spain was not included) is much better and improving. However the performance to East Europe is very variable, and the overall trend is deteriorating. The creation of TEN-155 is expected to improve the situation in both West and East Europe. Performance to Japan is good.

Figure 5: Transoceanic Performance from Sites in Europe

4.5 Transoceanic Performance seen from Europe

DFN, the German national Academic and Research (A&R) network, which provides connectivity for DESY, peers with ESnet at Perryman, a network exchange point in Washington, and CERN has a pipe to ESnet also peering in Washington. Hence the packet loss is low and the overall transoceanic connectivity to ESnet is good. DFN peers with vBNS at MAE-East, a heavily loaded public exchange point. CERN peers with vBNS at MCI’s hub in Dallas, which is better than a public exchange point, but it is still the commercial Internet. Consequently, performance to vBNS is not as good as to ESnet.

4.6 Transoceanic Performance seen from Asia

Transoceanic performance seen from KEK has been somewhat variable, although lack of statistics makes it difficult to look at long term trends. Performance has improved dramatically in recent months because of a new peeering arrangement with the European TEN-34 network.

Figure 6: Transoceanic Performance from Sites in Asia

Figure 7: Surveyor measurements

5 Conclusion

Overall, performance is improving. The question is, is ths improvement because HEP largely has it own network ? Performance on Academic and Research networks is good, performance on commercial networks is bad. Peering is critical.

6 Future Work

6.1 Surveyor

6.2 Netmaps

Network snapshots using traceroute.

7 Further Information

References

[1]

[2]