ESSC Meeting Washington January 21-23, 1998

Rough notes taken by Les Cottrell, SLAC


There were about 22 attendees. The room was wired for Ethernet. About 16 attendees had their laptops plugged in during the meeting. All the laptops were Windows based, most were from IBM, followed by Gateway and Dell.

Highlights & Important Issues from my perspective

ESnet futures etc. (see ESnet Issues & Futures Discussion)

PictureTalk demo (see Services).

ESnet international traffic (see International Links)

Internet 2 & ESnet peering update (see University Access)

NIMI/NPD tools (see DOE Research on Internet Measurement - Vern Paxson via MBONE)

QoS (see Differentiated Services for the Internet - Van Jacobson via MBONE )

DOE NGI plans (see DOE NGI Plans - Dan Hitchcock)

DOE LSN summary (see DOE Large Scale Networking Workshop summary - Bob Aiken)

Network Monitoring (see Network Monitoring Status - Les Cottrell)

Upgrading SLAC ESnet link (see Future Requirements)

Table of Contents

Highlights & Important Issues from my perspective *

DOE Report - George Seweryniak *

Interagency activities: *

ESnet Update - Jim Leighton *

New sites: *

University Access *

International Links *

Services *

Sprint Services Reprocurement *

ESnet Issues & Futures Discussion *

Discussion with Hitchcock on DoE's perception of ESnet *

DOE Research on Internet Measurement - Vern Paxson via MBONE *

Get a handle on measuring heterogeneity, sender/receiver the NPD experiment *

Building "measurement infrastructure: NIMI (w/PSC) *

Tcpanaly *

Pathchar *

Differentiated Services for the Internet - Van Jacobson via MBONE *

ESnet Program Plan - Stu Loken *

ESnet Advanced Applications - Sandy Merola *

DOE NGI Plans - Dan Hitchcock *

DOE Large Scale Networking Workshop summary - Bob Aiken *

Testbed Applications for ESnet - G. Chartrand *

ESnet Program Review - George Seweryniak *

ESCC Report - Roy Whitney *

SLCCC Notes *

Network Monitoring Status - Les Cottrell *

Future Requirements *

DOE Report - George Seweryniak

Handed out information on ESnet Review report. The DOE as opposed to the ESSC is now leading the production of this report.

Don Austin not coming (was going to lead the Grand Challenge effort) to DOE/MICS.

A new head of MICS will be hired to replace the acting head (Dan Hitchcock)

MICS now relocated in E-wing of 2nd floor of GTN office.

Interagency activities:

DOE got no NGI money.

Large scale networks:

MICS Initiative

ESnet related activities

FY 98/99 Programmatic goals

Lost $1M out of budget this year, Seweryniak hopes the review process will help the budget process. Merola pointed out that the review is only one aspect of information that will be provided, but the driving users who are being supported by ESnet is represented by the ESSC, so they should have the major impact. Feedback is needed to Dave Nelson from the program managers on the success of ESnet. The decision on ESnet's goals should be decided by the ESSC. The review committee should be given a charge so they can address it (i.e. the review committee should not decide the goals). So the question is who decides the goals, is it MICS or is it ESSC. A discussion is needed with Nelson, Hitchcock and Seweryniak.

ESnet & NGI programmatic goals:


ESnet Update - Jim Leighton

ESnet is moving 17.5Gpkts accepted, 8.23 Tbytes accepted (Dec-96 was 11Gpkts accepted). Data was lost between Dec-95 and Sep-97.

New sites:

Jim showed some MRTG plots of the traffic between ESnet and some of the 5 major ESnet peering points (Sprint NAP, MAE-West, MAE-East …)

University Access

ESnet has established peering with vBNS at SDSC, MAE-West & Sprint NAP. Perryman & Ameritech are coming online soon. Jim showed a list of likely peers that would allow access to a long list of universities.

wpe2.jpg (76005 bytes)

wpe3.jpg (69534 bytes)

CalREN2 is expected to be up late Spring. Expect one of the N. Cal GigaPOPs to be at UCB. Jim also showed a list of vBNS routes today (about 80, some universities appear multiple times).

wpe4.jpg (99659 bytes)

International Links

Most are reasonably loaded at the moment, number of complaints confirms.

On all international connections outbound traffic out outweighs inbound by factor 2-3.

wpe5.jpg (68356 bytes)

wpe6.jpg (68789 bytes)

They are looking at a more general routing policy to cover how to carry other people's traffic. ESnet will put separate W & E coast routers with a virtual connection over the ESnet ATM. This will carry the transit traffic. This contains the traffic (i.e. international transit traffic does not use the ESnet AS, international exchanges can be allowed on a pair-wise basis, access to/from ESnet is well defined), it has its own Autonomous System (AS) and the traffic can be constrained and so it is accountable.


Telepresence toolkit testbed.

Two examples of Telepresence tools are PictureTalk and an Audio Bridge (available by calling LBNL - Bob Aiken (who was not present in person at the meeting) was using both of these during this presentation (Jim Leighton's talk on ESnet) so he could see, listen and speak during the presentation). PictureTalk runs on Windows (NT & 95), Macs and several Unix flavors. The Audio bridge is at LBNL and has 24 ports available.

PictureTalk is a data conferencing tool allowing conferees to see the presenter's screen (works at screen level so does not depend on the application), the presenter's role can be changed during a session, a session is easily set up via the Web, and can be password protected, members use the web browser to view the presenter's screen, presenter selects portion of desktop to be seen by others.

Jim made his talk (in PowerPoint on a Windows laptop) available in real-time on the network using PictureTalk. PictureTalk can be used to share people's screens for example to see a presentation. PictureTalk is similar to Net Meeting. It is a plug in for your browser. It is available at This site included how to downloaded the plug-in as well as what talks are being conferenced. I (and about a dozen other attendees) loaded it and could see (and capture) the transparencies for Jim's talk. About 16 others also had PictureTalk connections to the presentation (about 3 were not physically in attendance).

ESnet provides the server. The client is free, they charge for the server (a year ago a 100 seat server was about $8K, see for pricing details). Audio support is available, but not yet sufficiently robust (it is half duplex, slightly delayed, often needs tuning). It can be used in conjunction with an audio bridge for higher quality audio support. It can also be used for a help desk application. PNNL have a similar system (TeleViewer, see - you need to register on the Web first) which adds data compression, the ability to draw on the presenters transparency. It is part of the COllaboratory Research Environment (CORE-2000) project.

The Audio Bridge has 24 ports, is very user friendly, it has a scheduler with date, time & length, optional password control, it can record the meeting and make the playback available and they are looking into web reservation capability. For attending it supports voice sign on, can announce attendees, do a roll all, has admittance support (lock, delete last, etc.), can arrange separate breakout sessions (may require in advance reservation).

Other tools include the VCSS (VideoConference Scheduling System), MBONE support, MBONE gateway, and multi-session gateway (from HEPNRC).

They are evaluating an enhanced MCU, an ATM-based desktop video-conferencing (high performance, but unclear how many Labs will have ATM support on their LANs, so support for 100Mbps Ethernet etc. is important) and a video server (e.g. session record & playback).

The enhanced MCU supports 40 virtual (128kbps) ports in H.320 mode, all ports are dynamically assigned, auto-configuration is done from the VCSS reservation data base, and has an upgraded Web reservation system.

They hope to put PIM into the production backbone routers soon (for MBONE). The gateway provides connectivity between ISDN & vic/vat. The Multi-session Bridge allows non-multicast capable workstation to connect to a MBONE multicast session.

They are designing a common scheduling web interface for all the toolkit items, based on the VCSS.

The ESnet page is being re-designed to improve navigation and simplify it.

Sprint Services Reprocurement

Have assembled a procurement team and have started preliminary team visits to MCI, WorldCom, Sprint & AT&T. They are evaluating alternative procurement approaches. They have a schedule that releases the procurement 2Q98, to begin the transition 1Q99 and to be complete by 3Q99.

Cost of bandwidth may come down due to WDM (several carriers are now deploying 16x WDMuxing and are looking at up to 40x) and new long-haul carriers are coming of age (IXC, Qwest, Williams who have between them an extra 35K miles of new long-haul communications by '99). In the meantime shortages in deployed bandwidth continue and many major ISPs have eschewed public exchanges for private 1: interconnects, and there is evidence that some of the private exchanges are now congesting without sufficient advance planning.

ESnet Issues & Futures Discussion

Program Office Issues

Future directions for ESnet

Role of ESnet committees

NGI Issues

ESnet review planning

Discussion with Hitchcock on DoE's perception of ESnet

The "lowering" of the MICS ESnet budget was actually a "relabelling" ESnet connection money to NGI and back. The ESnet budget has not increased at the rate of spending. There are questions in DoE about the number of networks (3) it is running. Having advanced requirements is a good defense about simply outsourcing the network to a commercial vendor who can provide commodity services, and separating the requirements of the DOE business network from ESnet.

Dan identified some major technical risks that the networks will need to address:

In response to a question from Larry Price, Dan stated that MICS feels the need to supply high performance networking to single purpose Labs such as FNAL & SLAC. Dan also said MICS might consider providing half circuit funding to a site such as CERN if it would act as the European connection point for ESnet (i.e. be prepared to forward traffic etc.)

DOE Research on Internet Measurement - Vern Paxson via MBONE

Get a handle on measuring heterogeneity, sender/receiver the NPD experiment

The Internet is very diverse - no single path is typical or representative

The key idea is to run a specially designed measurement process at a number of Internet sites

The NPD (Network Probe Daemon) runs at about 30 sites around Internet (~1000 links), accepts authenticated traceroutes, source/sink a TC probe, run tcpdump/rtcpdump to monitor a probe.

Analysis of the data could give an idea of the Internet in general. Included ~ 40K repeated traceroutes. Looks at pathologies, routing, stability (most paths heavily dominated by a single route, and also how long is a given route used - found 1/3 of routes changed over seconds to hours), & routing symmetry (if know route A to B how likely is I that reverse route is the same - at end of 1995 50% of routes visited at least one different routers in the path).

End -to end dynamics the SIGCOMM '97 paper analyzes 21K transfers (100KB) between 31 sites. They analyzed the pathologies: out-of-order delivery, replication, corruption (1 in 5000 packets arrive corrupted). The loss overall nearly doubled 1994-1995. The loss is far from independent. Distribution of the outage duration has infinite variance.

There was no single time-scale associated with congestion (rule of thumb 100ms to 2 s). Experimentally assessing available bandwidth was difficult due to competition for the link.

Frequent path asymmetries were seen in delay, loss, bandwidth.

Building "measurement infrastructure: NIMI (w/PSC)

This is an NSF/DOE supported pilot project to develop a "measurement infrastructure" for the general Internet The idea is to deploy inexpensive measurement platforms for diagnosing performance problems, baseline measurements for understanding traffic, assessing performance delivered by IP clouds (don't do it themselves, rather make it possible for someone else to do)

NIMI supports a wide range of active measurements, it is designed to scale to >> 1000s of platforms (avoid success disaster), platform owners have (potentially) full admin/policy control, solid security designed from the beginning, and requires minimal administration (does maximal self-configuration).

The basics are to provide the ability to schedule future participation in a measurement, pick up results, delete measurements, cancel or suspend a schedule, query the current schedule.

The security model is that requests to a NIMI platform includes the name of a credential and a corresponding cryptographically-secure signature, authentication is based on ACLs, rows correspond to credentials, the columns correspond to actions, you control access to a platform by controlling who has the corresponding credentials (requests are also encrypted, prevent man in middle attacks) and it is implemented on top of RSAREF.

For auto-configuration, each NIMI platform has one master point of contact; the master can modify the entire ACL table. The master downloads the kernel of an ACL table and redirects the platform to other configuration sources. When platform re-initializes it request the configuration from the master.

Standard platform is <$3K (200MHz, 64MB RAM, 4 GB disk, 10/100 Ethernet, modem, and an optional GPS). It is built on FreeBSD. Perl prototype is running on PSC, LBNL platforms. Includes auto-config, needs GUI.

Future plan to deploy at DOE sites (in collaboration with HEPNRC & SLAC), longer term deploy by volunteer ISPs (e.g. WorldNet). They want to orchestrate measurements and provide an analysis framework. They want to archive results, navigation, and visualization issues, want to integrate with PingER, Surveyor.

The research issues include: self configuration (where am I logically in the world, the mapping problem); integration with HOPS, going from individual results to problem diagnosis, aggregating results into higher level statements about regions. The Holy Grail is an end-user clicks on a button to diagnose a problem, this queries the distributed database and maybe causes extra measurements to be made.


Tcpanaly analyzes a tcpdump, or pair of traces. It detects measurement errors such as busted clocks & packet filters. It knows the behavior & foibles of 9 TCP implementations (SunOS, Solaris 2.3/2.4, HPUX 9.05/10.0, Digital OSF/1, IRIX 4.0/5.1/5.2/6.2, BSDI 1.1/2/0/2.1, NetBSD …). It separates TCP behavior from network behavior. The goal is to provide network and transport analysis to aid in understanding end-to-en application performance.


Helps find path characteristics, is similar to traceroute but designed to isolate congestion and loss problems rather than routing strangeness. It determines the bandwidth, propagation delay, queue & loss rate for each hop on an Internet path. It uses the same mechanism as traceroute (i.e. modulates the TTL & looks for ICMP "time exceeded" messages), works wherever traceroute works.

It provides estimates of delay times, the length (in time of the queues), the optimal TCP window size to match the performance of path (especially important for inter continental links), the percent packet loss.

Differentiated Services for the Internet - Van Jacobson via MBONE

Giving better service to some traffic means worse service to the rest. Despite ATM marketing fantasies it is a zero-sum game. A major issue is who controls the policy (is it users or institutions). If users the solution is trivial since there are no trust or coordination issues. Unfortunately this doe not work since it assumes there is always enough bandwidth to handle all users' needs. In a world of finite bandwidth, institutions have to control the sharing. Since users can't get whatever they ask for you need good security so users don't lie.

What are the target applications, in 1978 it was RJE, in 1998 was email/FTP, in 1998 probably the Web, this too will change. IP/TCP/UDP/IGMP/OSPF/BGP work for any application, and differentiated services must also work.

What is the service: better best effort or virtual leased line? We need both of these services. The former is being pushed by the ISPs so they can service the big money customers when things get bad. The accountants at the customer sites however want virtual leased lines; it is more of a quantifiable entity.

A differentiated service mechanism must work at the scale of the Internet (e.g. millions of networks) and at all speeds so must push "state" information out to the edge. "Edge tagging" has been proposed as a way of offering differentiated services, the leaf router adjacent to the source(s) has traffic signatures for special traffic and a profile maker giving its characteristics, leaf router marks the packet. Then the site border router (on the ISP side) allows marked traffic to the amount purchased in an agreement by the site. There is still the issue within the site as to which users can get the special service. In essence the user requests go to the organization which accepts and OKs the requests based on organizational policy. This is done by the bandwidth broker (BB) to be the repository of policy database of priority and limits for the user & project access to the special (marked) bandwidth. The repository includes credentials. The BB provides information to the leaf routers to allow them to mark the traffic. The ISP then accepts up to its contractual limit, if beyond, it throws away the excess in a random way, so the site loses traffic and the pressure to fix things is pushed back to the site.

Design constraints include: inter-domain service, the end-to-end service should be constructed from concatenated bilateral agreements, the service must not require extending the trust or control across administrative boundaries (upstream can't force extra work on downstream), upstream must not cause problems for downstream.


If no shaping is done, burst size scales like 2**N for a source N hops away, if shaping is done at borders, burst size scales like average in-degree in region. With no shaping, the system is very brittle (premium users may see very poor service depending on their location in topology and/or competing traffic). However routing vendors do not do shaping (apart from some pre-production stuff from Cisco) partially because in order to do well in the "Bradner" tests they need to get packet delivered ASAP.

Requests have to be made end-to-end so it has to be done at the border routers along the path and only if all border routers agree to accept, is the request fulfilled.

ESnet Program Plan - Stu Loken

Stu had handed out a draft earlier, for reading (I have a copy). Comments should be sent to Stu or to This will be an important component in the upcoming review. The upcoming review in May will focus on the future strategy rather than operational issues.

ESnet Advanced Applications - Sandy Merola

A draft copy was handed out in advance of this presentation (I have a copy). The outline is that: it commends ESnet on its success, identified network application areas that will accelerate DoE science, it encourages ESnet to be responsive to the programmatic applications community as well as the network research community.

The application areas are: remote experimental operations, distributed parallel computing, remote/shared code development, remote & distributed data access, visualization, and tele- and video-conferencing.

The report also looks at the opportunities and requirements that are driven by anticipated service-offering advances into the network cross-cutting areas of: quality of service, non-backbone connectivity, network status & diagnostic tools for users, scalable communications, network and application security.

The report recommends MICS-funded network research and development efforts to create/enable high performance networking applications. ESnet should support this work. In order to minimize the impact on the production ESnet network the issues raised in must be considered.

Seweryniak would like it to be included as part of the ESnet program plan.

DOE NGI Plans - Dan Hitchcock

The FY99 President's Request is focussed on:

The core includes: 2 testbeds (Diesel and MMC), technologies (CBQ, multicast, secure access, high-speed interfaces, monitoring).

The enhancements needed include:

Looking at the relationship with universities and how to improve, the plan is:

The way the partnership is proposed to be improved is:

Dan hopes the chances of success are better than last year. He feels there is more focus, and better planning. There is supposed to be about $12M extra money. DOE hopes to get 33% to 50% of this if successful. He hopes the matching money put up by DOE will not be at risk.

DOE Large Scale Networking Workshop summary - Bob Aiken

Bob feels the Networked Challenged Applications failed miserably. When I questioned him more closely on this he broke it down into 2 reasons:

Bob says they will probably fund some of the stuff and gave as an example adapting HPSS since this is building an infrastructure and providing middleware.

The goal of the LSN workshop was to identify & define new network capabilities, technical & infrastructure needed for enabling new revolutionary DOE apps. A 2nd goal was cross-pollination between ER & DP (network researchers, middleware researchers, application researchers.

Technology topics included HIPPI64, QOS/CBQ, multipath, and multicast reliability.

Identified research topics:

The above implies resource management, accounting/costing, policy management, prioritization & preemption for resource sharing, PKI, multicast, virtual dynamic networks (a la Morphnet)

Candidate testbeds:


MICS Future R&D Directions

This is being driven by the perception that DARPA is the research agency, DOE is more of an applications agency.

Testbed Applications for ESnet - G. Chartrand

Greg chaired a working group to identify possible testbed applications for ESnet. The following testbed items were put forward:

Greg also added the testbeds suggested at the DOE LSN workshop by Bob Aiken


ESnet Program Review - George Seweryniak

This is due in May at LBNL. Larry Price will help facilitate. George wants the charge, and suggested participants. Previous reviews were in '89, '91, '94 and now '98. Actions from the '84 review were to develop a strategic plan, an annual operating plan, interconnect networks, provide consulting services for the community (this action was not accepted), and improving public relations (lot of education, wider distribution of ESnet publications).

Suggested title is "Future Direction for ESnet - Past, Present & Future"

This is a strategic review vs. a tactical review, the issues to review are:

Potential reviewers

ESSC will be intimately involved in selection of reviewers, the charge, they will provide the site, and help facilitate. Pre-requisites: complete program plan, complete progress report, complete advanced application document, complete charge for reviewers, finalize & contact reviewer, prepare and send packages to reviewers.

The questions for the Review are:

The following cross cut issues were identified

  1. External
  1. Internal
  2. Resources (internal)
  3. Planning

Roy will explain this

The following were suggested as members of the committee (they want a total of 7 including the chair):

Outline of agenda

It is 2 days (of the May 4-7, 1998 ESSC meeting), 1.5 day of presentations.

ESCC Report - Roy Whitney

Next meeting Berkeley 23-27 March.

The last meeting was at CEBAF. In the networking the main issues were IPngWG, & Network Monitoring.

IMAP servers are becoming more dominant in the industry and at the Labs. Another big issue was security. Four Labs (SNL, KCP, LLNL, LANL & DOE-HQ) are fully cross-certified and LLNL is cross-certified with NASA. Three more Labs including ANL & PNL are coming on soon. The DCE group has had considerable success. The DOE business net is being picked up by HR, who are hoped to be easier to work with. It costs about $1M/year to run. Running the DOEBN as a VPN over ESnet might cost about $100-200K / year. The SLCCC has sent a letter to the DOEBN.

A new ESnet progress report is close to completion. It will be available online next week. A preview is available at The leaders of the various sections will be asked to review & provide comments on the section. The final version will be ready in final version in March, and will be in Web format


DOE is becoming concerned about the Y2K problem. They have set up a security working group (Tom Nash is setting this in motion)

"Roadmapping" has become a big buzzword at DOE. The idea is that one defines the user goal (e.g. find the Higgs), then one figures out what the critical research & development and management that is needed to reach the goal in the optimal fashion. This then defines a set of R&D etc. goals that need to be reached and this is the "roadmap".

Network Monitoring Status - Les Cottrell

The talk can be found at: Following my 45-minute talk there was a 30-minute discussion. There was unanimous support for continued funding of the effort. In particular Walter Toki, Bob Woods, Jim Leighton, Bob Fink, Larry Price, Martin Greenwald, Sandra Merola and Roy Whitney spoke in support of the effort. Jim Leighton said the tools are complementary to the tools ESnet uses, he uses our tools and fully supports them. Bob Fink said that these tools fit in the rubric of infrastructure/middleware tools. There was a discussion of how to make the results better known at higher levels and more public. Martin Greenwald was concerned that we (DOE) often come up with useful tools but then fail to fund the deployment, it is like as soon as it is determined that something is useful we drop it, let's not fail in this case.

Some suggestions were made for how to summarize the information at a high level. Roy Whitney was asked to get together with Les to come up with a high level summary (e.g. International links get saturated with some frequency, vBNS is helping but possibly not as good as a direct ESnet connect). Chris Witzig (BNL/NP) will work with Les to provide a list of RHIC sites (for the grouping).

Future Requirements

Les Cottrell and Walter Toki requested to upgrade the SLAC ESnet connection from T3 to OC3. This is driven by emerging BaBar requirements in particular the need for excellent connections between SLAC & LBNL and SLAC & LLNL. These links are needed to allow the use of remote computing cycles at LBNL/NERSC and LLNL for Monte Carlo generation and for reconstruction. Besides needing the higher speed links to transmit the Monte Carlo and reconstruction data, they would also be used for code building, interactive analysis. SLAC has made tests (using OC12 NTONC links) between SLAC & LBNL to show that sustained rates of 200Mbps for TCP and 400Mbps for UDP are possible.

It was agreed that SLAC/BaBar would prepare a case for the upgrade for the next ESSC and Jim would look at the costs of such an upgrade. We (SLAC) also need to work with NERSC (Horst Simon) on this so it can become a multi-program requirement, which is easier for MICS to support.