Southwestern Bell, Austin TX, 18-19 Sept 1997

Trip report from Les Cottrell, SLAC


Welcome - Howard Shimokura, SBC *

Internet Weather Report (IWR)- John Quarterman, MIDS *

Emerging Measurement Tools & Initiatives - Tracie Monk, NLANR *

End-to-end Methods for Measuring & Improving Internet Performance - Jeff Sedayao, Intel *

Technical Subcommittee T1A1 on Performance: Status & Plans for Internet-Related Work - Spilios Makris, Bellcore *

Metrics & Infrastructure for IP Performance - Guy Almes, ANS *

Internet End-to-end Performance Monitoring, Methodology, Tools & Results - Les Cottrell, SLAC *

The Automotive Network eXchange Overseer Performance Test Tool - Dorata Blat, Bellcore *

Testing for Structure- John Leong, Inverse *

Discussion/Planning - Ira Richer *

Discussion of what to do next? *

This was meeting of the Cross-Industry Working Team (XIWT) Internet Performance Working Team (IPWT). It was a very interactive meeting with about 25 attendees with 8 laptops deployed. As far as SLAC is concerned, the most important section to read is the last one (Discussion of what to do next).

One outcome of the meeting was that the SLAC/HEPNRC tools for end-to-end performance monitoring have been selected to be deployed at about 6 XIWT member sites including (if my memory is correct CyberCash, SBC, Intel, Houston Associates) with Intel acting as the Archive site. They want measure performance of members' own networks, get some tests up to validate and understand what should be recommended to other commercial customers and for what purposes. A goal is to build a community within XIWT so can evolve it to address harder issues.

Welcome - Howard Shimokura, SBC

We are moving from a circuit switched world to a packet switched one. The Telecommunications /Information industry contributes ~ 16% of the US GNP. In the rest of the world it is typically < 6%. Thus it is a big contributor to the US economy and also the US has leadership so can export to the global economy. Performance monitoring is critical to provide users with a predictable and stable environment that they are looking for.

Internet Weather Report (IWR)- John Quarterman, MIDS

John gave an interactive demo of the many reports that the IWR has. Many of those demonstrated are only available for paying customers. He has been gathering data since 1993 and has come up with some very nice ways of presenting the data by graphs generated by Perl scripts. One of the graphs showed the Internet response time has improved from 650 to 450ms (averaged) between 1993 and Sept-97. The graph also showed distinct seasonal patterns. He also showed graphs of the 25%, median, 75% and the minimum (all plotted on one graph as shaded areas) for various metrics, which were intelligible and clear. One interesting graph was the number of hosts per domain versus time, with lines for all the domains (many tens possibly over a hundred) which was still intelligible. We talked a lot during breaks about how we might collaborate since we have critical needs on how to display long term data from multiple sources for many metrics. He is very interested in historical data and I will get him in contact with the folks at RAL who gather the traceping data, in case they have some long term data.

Emerging Measurement Tools & Initiatives - Tracie Monk, NLANR


For a copy of this talk see http://www.nlanr.net/Caida/xiwt_970918.html/index.htm.

Why measure:

There are different reasons for ISPs vs. "Users"


Required Tools

NIMI (National Internet Measurement Infrastructure)


As a group we need, for the health of the net, to encourage enabling some things such as reverse traceroute servers, identify ping hosts, provide trustable latitude and longitude of routers in the DNS records for visualization. This is needed from the ISPs. However, some of the ISPs are concerned with identifying the exact location of a site due to terrorist threats. For most visualization accuracy to a city or phone exchange or Zip code is good enough. Maybe as users we could ask for ISP to provide such open-ness in our contracts.

Flow tools:

Management tools


Note that NLANR/CAIDA tools are public domain

End-to-end Methods for Measuring & Improving Internet Performance - Jeff Sedayao, Intel




Internet Measurement & Control System (IMCS)



Technical Subcommittee T1A1 on Performance: Status & Plans for Internet-Related Work - Spilios Makris, Bellcore

Spilios is vice-chair of the T1A1.2 (732-758-5640)

T1A1 develops & recommends ANSI standards & technical reports on speech, audio, data, image & video & multimedia integration within US telecommunications networks. Also fosters consistency and positions in other North American & International standards bodies.

The motivation is the explosive growth of public internet services which is raising fundamental questions on standardization efforts. Performance measures for end-to-end Internet measurements have not been standardized. Delay, information loss for "best effort" Internet services are of increasing interest, as are also new services such as multimedia, streaming, inter-working with the PSTN.

T1A1 has a key role in harmonizing ANSI standards with the IETF and ITU. There are several working groups.

See www.t1.org/t1a1/t1a1.htm for updated information on working group activities. They have only just started and have not done anything substantive yet. The participants are mainly coming from the PSTN side. They are looking for improved liaisons between XIWT/IPWT and the T1A1 working groups.

Metrics & Infrastructure for IP Performance - Guy Almes, ANS


Objective is to enable users & service providers to have common understanding, tools etc.

IETF/IPPM have produced: a framework document, a one-way delay metric, a packet loss metric, bulk transfer capacity, and are working on an availability metric (Paxson & Mahdavi started Dec-96)

Measurement Strategies:

Framework revisions:

Surveyor Infrastructure

Database/Web server


There is limited analysis / reports available at the moment. He showed:

Policy implications:

Internet End-to-end Performance Monitoring, Methodology, Tools & Results - Les Cottrell, SLAC

See: http://www.slac.stanford.edu/grp/scs/net/talk/xiwt-sep97/

The Automotive Network eXchange Overseer Performance Test Tool - Dorata Blat, Bellcore

TCP/IP based VPN interconnecting automotive industry trading partners. They have 15K trading partners in N. America, and 50K worldwide. It is characterized by controlled service quality for mission critical business to business trading partner communications. The idea is to replace a special purpose network consisting of leased lines with an equally good (or better) performing network at lower cost. It comprises multiple ANX certified ISPs (CSP) & ANX certified exchange point operators (ANX CEPO, these are the folks who interconnect the CSPs).

ANX Service Quality is a metrics-based approach. There are 8 categories: net services, interoperability, performance, reliability, business continuity & disaster recovery, security, customer care, trouble handling. Bellcore designed the metrics, criteria & measurement techniques. Bellcore has the role of the ANX overseer and defines certification criteria for service quality, assessment of service providers, certification verification of ISPs.

The metrics are driven by trading partner quality needs. They have a test methodology based on a black box (they require their CSP to install these black boxes) approach which measure metrics visible to the end users (they are not interested in internet trends). These include file transfers, with test data of incompressible, unpredictable random data, and qualitative criteria representing bounds on acceptable performance. They measure throughput, packet loss & delay and the metrics will be enhanced based on ANX release 1 pilot experiences (Sep-Dec 97). The measurements are not done continuously but rather made occasionally (maybe a few times a year, or if complaints are made) on a sample basis.

The throughput target criterion is to measure throughput equal to or exceeding half of the access link bandwidth adjusted for capacity consumed by link-layer. Size of the test files 30+ Mbytes (their users mainly used large file transfers of CAD data rather than a lot of WWW browsing), size of IP packets 512 byte payload (avoids fragmentation). The throughput is averaged over several tests with a sliding window over the most recent tests.

The packet loss rate criterion (PLR = (# packets sent - # necessary packets)/#packets sent) <= 0.03% initially. The loss depends on link utilization. If the end node link is getting < 30% loading then the PLR threshold required is 0.03%, for 50% loading they threshold is 0.05%, if the loading is > 70% they throw away the measurement and try again later. They are likely to be made more stringent when running smoothly. The number of packets aggregated must be statistically meaningful (100/PLR). They put test points at the end of the relevant links. Modern, stable TCP performance is based on some packet loss to help identify how big the windows can be made before congestion occurs. This is especially the case on long high bandwidth links where many packets can be simultaneously in transit on the link. The question arose as to what percent packet loss is normal (i.e. caused by having a larger window size than the intervening queuing elements (routers) can buffer) for normal good TCP performance.

The file transfer delay criterion takes into account file size, access link rates, number of hops, propagation delays (physical distances). The measurement technique uses 1 Mbyte files, of 512 byte payloads, with a calculation aggregated over several tests with sliding window over the most recent trials. They demand that 90% of the file transfers meet the delay requirements.

Testing for Structure- John Young, Inverse

Describing ISP benchmark report to understand end user experience. They have done controlled tests on accessibility to 25 ISPs with 40 POPs (Point of Presence) per ISP to look at Internet performance in DNS, Web latency, throughput, loss. They enter the Internet from over 1200 locations. Need a standard test page, standard server, a basket of 10 popular URLs which all ISPs tests. Other issues that affect the user experience include the server, network, browser, modem. Then there is network caching &/or compression.

Their test setup can dialup any of the POPs and make the tests from there. Thus the test link can be a very poor compared to the backbones, it works OK for packet loss. Throughput is more challenging since limited by the dialup link. Very excited by pathchar since it shows one can measure the throughput of a fat pipe from a much slower link. By comparing various pairs (e.g. Netcom-MCI vs. Netcom-UUnet vs. Netcom-Concentric) they can start to identify a bad ISP. They use UDP transfers. They randomize in time the measurements, but are not doing Poisson distributions. The multiple monitoring points allow better understanding of where the problems lie (e.g. if monitoring to a site works from any monitoring site, the the remote site must be reachable). The destination node is co-located with the monitoring site so the clocks are fairly well correlated.

They find that inter ISP links are worse than links that stay within an ISP. Typically within ISP they see 0-4% for 90% of the links, for inter ISP links (which may go through several ISPs) they see >5% probably 10-20% of the time.

Discussion/Planning - Ira Richer

The purposes are:

Need to identify what is important to be measured?

Identify what will/could ISPs provide to the users (especially if it is in an SLA).

What measurements would one want the user sites to provide, or are useful.

There are lots of interfaces, data format, kinds of data being collected. Need to agree on a common data format, or common tools.

The IPWT does not want to develop tools, so they are interested in using existing tools. Most felt the tools demonstrated earlier were an existence proof. Some tools are based on extremely simple tools in particular ping and traceroute. A way of doing it would be that IPWT members become collection/monitoring sites and identify the remote sites they are interested in, the XIWG provides the analysis site and builds the tools.

There may be a sensitive issue with a standards body (such as the XIWT/IPWT) identifying with rating products. This would be especially so if the metric is subject to optimization by the ISP, e.g. GETting Web pages at a few well-known sites, so the ISP identifies those sites and provides good connectivity to them. One possibility might be to define the tests, but not publish them. This also reduces possible competition to companies like Inverse. Could publish but not produce conclusions, or publish the results without labeling them (i.e. making them anonymous). Tracie Monk claims that, even if don't label the reports, some experts can identify from patterns which ISPs may be involved. This may not be a problem if the number of experts is small.

Another issue is long term trends versus real-time monitoring for trouble shooting especially for the site NOC. Both were felt to be of interest.

XIWT/IPWT maybe a good group to work on hard collaboration (in Guy Almes definition of hard vs. soft collaboration) since they consist of a group of members who are interested in performance monitoring.

Another concern is how to do analysis/presentation so people may not be misled. This is where much of the work is. Maybe some commercial outfit can mine the data so they can sell the results to customers or ISPs.

Much of the discussion was on what should be the role of the XIWT/IPWT. Two possible roles came up:

  1. "confront" the ISPs with performance issue with an interpretation asking them to address the issues. Maybe do it via IOPS.
  2. A second way would be to provide tools for ISPs and others to use.

How does one want to promote measurements, should the XIWT be promotional, and who should it promote, e.g. the IETF, the T1A1, ANX efforts. John Quarterman pointed out that one of the things that promoted TCP vs. OSI, was the defense folks bought a lot of Sun workstations, so an analogy would be for the XIWT to buy measurement services (John acknowledged that MIDS is a major provider of such services and so he was not unbiased J ).

Discussion of what to do next?

Want to measure performance of members' own networks, get some tests up to validate and understand what should be recommended to other commercial customers and for what purposes. A goal is to build a community within XIWT so can evolve it to address harder issues. The XIWT represents a very different audience than the SLAC/HEPNRC, Intel, Surveyor communities. In particular the XIWT includes ISPs (e.g. GTE, SBC, AT&T), Internet providers (e.g. CyberCash), commercial users (HP, Intel).

What tools should be deployed?

Possibilities: SLAC/HEPNRC, Intel, Surveyor (Almes), NIMI. There are other tools such as Merit and flow monitoring which provides traffic flow characterization. Flow monitoring requires a higher level of trust and cooperation. It is easy to monitor own flows. There might be value in aggregating data from the flows of multiple users. Flows also have severe privacy issues.



Stability, maturity, what does it measure, underlying tools, is it still evolving, is it supported, what are the resource requirements (people, hardware, system etc.), visualizing the data, upgrade plans, ease of use (install, documentation, consulting).








Packet Loss, RTT, unreachability

Packet loss, RTT, HTTP GET, DNS

1 way delay, packet loss

Delay, packet loss, traceroute, throughput

Underlying tool




Treno, traceroute, ping


Plots of loss RTT vs. time, Web tables of most recent measurements with navigation. Pilot of long term tools





Flat files & SAS

Flat files




0.5MB/ping pair










SAS (archive site) for NT/Unix





DIY and is documented, root access not required

Intel does installation, requires root access




0.3 for archive site, 0.1 for collection site




Upgrade plans





Level of support available





Ease of use

Some, man pages, attention has been paid to configuration, instructions for installation




Firewall preference

Can be inside or outside, currently it is usually inside the firewall


Can be inside or outside

Prefer outside since uses dynamic UDP ports

Possibility would be to install SLAC/HEPNRC or the Intel tools generally among the XIWT community, with a few more aggressive sites (a sub-team) pursuing the Surveyor. The existence of SLAC/HEPNRC's visualization tools is a very valuable selling tool to management. It may depend on how soon results are needed. Also it may depend on how important it is to have one-way timing. Can XIWT members do both? About 6 XIWT members are interested in deploying tools. There was a move to start out with the SLAC tools, but not exclude Surveyor to later use. Could buy a Surveyor box and run the SLAC/HEPNRC tools on it. Note Guy Almes (father of Surveyor) wants to put Surveyors at a few DoE sites including SLAC/HEPNRC so there should be a good growth evolution.

Next steps/Action items

Next meeting:

Is there interest in developing a model contract for ISPs? The subcommittee will jump start this.

There was a recommendation that the collection sites provide a reverse traceroute server at their sites.