ESnet International & ICFA-NTF Meeting

Santa Fe, September 8-10, 1997

Rough Notes Taken By Les Cottrell


International Status

ESnet Update

Report from ICFA

Questions Raised by Working Groups

International Nuclear Safety Program

Introducing Submarine Cables

Satellite Update

Video Conferencing

Distributed Databases



ESnet International Issues

Transatlantic Requirements

Requirements Working Group

ICFA-NTF Discussion

Interim Report Discussion

Global Networking

TEN-34 - Fernando Liello

TEN-34 see, for DANTE see

Mixture of ATM & leased circuits

Combination of managed services & IP links

ATM policy is only academic & research

IP includes commercial on best effort basis

Work item for international connectivity, but no funding

Economics of sharing are improved by TEN-34

Currently 68Mbps NORDUnet

16Mbps SURFnet

53Mbps Janet

TEN-34 Progress

2-8Mbps CERN/Switch

50% costs for same capacity

Loading Paris London (only 6Mbps ATM) also Netherlands-Germany (22 Mbps) 40-50% loaded

London-Stockholm 20-30% loaded

In general ATM lightly loaded, IP is heavily loaded in some cases, partially due to commercial traffic


TEN-34 commitment only until Aug-98

Too many suppliers, too expensive

Liberalization in Europe Telecomm Jan 1 1998.


US access restrictions do not match European policy

Different US discipline orientations

Limited funding



Europe-US 305 Mbps

Europe - US lines $40M / yr

NSF contribution $2M?

Funding question remains

Hard to justify

NSF Solicitation

DANTE part of University of Illinois submission

Will utilize existing lines & TEN-34


GAP in funding remains

ESnet contribution ($100K/yr for Italian co-funding)?

US political will to solve the problem?

DFN Status - Martin Willhelm

WIN (broadband network) in operation for more than a year. AUP does not allow commercial transit. Has a running ATM backbone used for transmission of IP traffic. Have 10 ATM nodes with 350 institutions from research & education. Have got more sites connected than expected. 10 sites at 155Mbps, 64 at 34Mbps. Overall performance and stability is quite good. Contracted to Deutsche Telecomm. Still have a small part of X.25 network to be phased out next year (it costs considerable money, and some users are hard to move). Will be offering native ATM connections. Have some pilots running. Hope to offer at end of next year to some universities. Have requests to put voice on. Preparations for next phases have started. In next 2-3 years hope for transition to OC12 network. Have pilot running today. Have TEN-3 connection at 34Mbps which is quite satisfying. Currently carrying 34Mbytes/month. They have a guarantee that up to 400Mbps peaks they will have good performance. They now carry 300Mbps peaks, so may expect problems in 6 months. 

US connection at Perryman (MCI clocation space) , MD with 90 Mbps (2*T3 on different cables, one on PTAT-12, one on PTAT-13, cost is about 22M DM/yr, i.e. about 1/3 cost & bandwidth from Europe to US). Deutsche Telecomm manages 2*Cisco 7513 routers at Perryman. They have redundancy (2 routers, 2 switches) also can dial in out of band, as well as a contract with MCI to provide technician support. Have transit agreement with MCI to US and rest of world. Have a T1 connection to Esnet. Will be upgrading to T3. Have rented additional space for other parties such as DANTE so they can colocate there also. Not considering a presence at STAR-TAP, no need, can't afford. NGI considering 3 interconnect points (W Coast, Chicago, E Coast), so DFN could connect to E Coast.

Italy - Antonia Ghiselli

GARR-B follow on (Broadband)

Requirements for US connections

Need 4Mbps or 155Mbps with peering to ESnet, NASA, Internet 2, IP general Internet, G7/GIBN, others.

Looked at Telecom Italia ordered 45Mbps on Columbus 2 ordered July 97, delivery ?)

Via TEN-34: 10-20Mbps thru Milan-Franfurt (0ct-97)

Bunch of 2Mbps

Proposal for US link:

Japan - Yukio Karita


KEK & most universities on HEPnet-J, using PVC

International lines

HEP dedicated KEK-ESnet 1.5Mbps last Saturday very good

KEK-IHEP upgrade from 64kbps to 128kbps

KRK-BINP 128Kbps also slowly progressing

NACSIS general A&R lines

NACSID - US SprintLink 6Mbps (heavily saturated, very bad, unusable) -> 45Mbps line

NACSIS Europe (DANTE) 2Mbps becoming bad

Two major programs for US & TEN-34

US universities

Lot of complaints from researchers in US universities

Have been expecting routes to US universities to be given thru Esnet. They wll stop using

NACSIS-US line until upgraded to 45Mbps so get good connectivity via Esnet. TEN-34 has not been good for Japan. Had assumed termination of NACSIS-US line would be moved from Europanet to TEN-34, but was wrong. Europanet link goes thru saturated link Geneva - London. Would like to move termination of NACSIS line to TEN-34, but Unisource insists on charging a fee for the access of $34K/month. Note that NACSIS is already paying for the full circuit if the NACSIS/DANTE line (Europeans pay nothing), also note DANTE does not charge for connections. Thus NACSIS is relocating termination point from Amsterdam to London to avoid paying $34K/month. But relocation takes time. Afraid London-Geneva gateway will be saturated before the relocation is done.

UK - Paul Jeffreys

UKERNA (an independent government funded non profit making company) operates a 155mbps national academic network.

UK Europe

TEN-34 service started on 18th March

Added parts of TEN-35. RTT halved, packet loss much better

Performance to CERN is bounded to deteriortae

Conutries are committed to cancelling their rented lines


1995 4Mbps for entire UK

Early 1996 8+6 * 1.54Mbps)

Mar 1997 t3 (45 Mbps) WWW traffic tken onto old connection

Mid july - early Aug 1997 fiber dies had to use satellite.

Harvey says up to 85% utilization packet loss is small, then starts to rise

Canada - Dean Karlen

CA*net has a linear 10 Mbps link with 4 links to US (aggregate of 90Mbps). One of which is a direct link to Esnet to bypass MCI problems (e.g. at West Orange).

CA*net II will use OC3 with GigaPOPs at Province capitals. Has connection to STAR-TAP. Does not replace existing commercial networks, organizations must maintain a separate link to the Internet. <10% of University traffic will be diverted to CA*net II

AUP Tier A: Universities & Labs. allow transit traffic. See STAR-TAP as an international exchange point.

Very interested in full peering at STAR-TAP, expect all major research nets will meet at STAR-TAP.


They are getting good connectivity via ESnet to Germany & Japan. Would like to improve connectivity to Russia, in particular the Khurchatov institute in Moscow. ITER ends July-1998. Have not selected a site yet. Hopefully then can move into a construction stage. May enter a 3 year pre-construction phase. Target is by 2009 many countries can connect to the ITER database and use interactive control.

DESY - Michael Ernst

DESY has 34Mbps to BWIN. Still has connections to CERN link, cancelled mid '97 and connect via TEN-34, and terminate wide-area DECnet. US/CA connectivity via 2 * 43Mbps links.


International is satisfactory to US/CA, excellent over TEN-34

US/ESnet 2-10%, US/CA 10-20%, Japan 5-10%, CERN/CH 0-5%, UK (RAL) 0-5%, FR (CEA) 0-5%), Germany 0-1%.

Vol.ume DESY-International 180GB, Int.-DESY 150GB, national traffic is similar. Leighton (using pathchar) sees losses are occurring over transatlantic link (from US to Germany). Michael showed graph (created using the MSRP tool that looks at router statistics - several others showed graphs using this tool which looks extremely useful for measuring utilization) showing that transatlantic link is saturating for several hours each day.

In Hamburg have an FDDI MAE network, will be changed over to a central ATM switch network.

ESnet - Jim Leighton



International Issues

International connectivity on East Coast

Upgraded ESnet DFN, bandwidth, cost & bandwidth guarantee TBD.

CA*net T1 to PPPL may be obsoleted by ESnet connection to Chicago NAP, interest in access to CERN.

CERN carry as ESnet primary site?

Connection to Caltech underway (executive at Caltech will write letter).

DANTE plans to establish a POP in Perryman (good news).

General university access (needs written letter from high up in University stating that they are cognizant of the special connection and nobody on the campus has a problem with it).

STAR-TAP access is coming.


There are only 16 Universities with vBNS access today. Jim was unable to get a schedule of when the other 30 odd sites will be added.

VBNS moving ahead with Internet 2 and connections program.

Bad list of Universities from ESSC study was (+ already there, - soon (we hope)):

The poor list was:

Report from ICFA - David Williams


David showed a lot of SLAC generated 180 day plots of performance and narrated the observations. He said they were extremely useful to understand what is going on. These were presented at the Hamburg ICFA meeting.

Questions raised by Working Groups

Monitoring WG

  1. How to cope with all the data, make it intelligible
  2. Need to add analysis effort
  3. ISP relationships, how to work for trouble shooting & long term improvements
  4. Code development
  5. Next steps

Requirements WG

  1. Satisfy the needs within a reasonable budget
  2. Consequences of not satisfying needs OR what science will be enables OR when do present computing models break

Status WG

  1. Bps to major HEP sites
  2. Normalize to traffic of National Research Networks (NRN)
  3. Calculate cost of HEP network usage

Remote WG

  1. Country contacts in HEPNRC database
  2. Define cutoff/data relevance

International Nuclear Safety Programs - Greg Chartrand

New program:

Big challenges are coordination, contracting, customs, communications, travel, building trust, demonstratin relevance, fulfilling commitments, achieving safety.

Greg is looking for collaboration and exchange of information.

Introducing Submarine cables - David Williams


A major limitation (on number of pairs and installation cost) is for the repeaters. Underlying costs are halving every 18 months. Expect factor 100 in next 10 years. There are beginning to be non-consortium run cables appearing, maybe this will bring down costs due to competition. There appears to be a big markup (10 to 20 times) of cost of installation against what cable owners sell capacity.

Satellite Update - Hans Frese

Two main uses of satellite capacity:



C Ku Ka Band

4/6 12/14 20/30GHz CF (only C band is rain tolerant)

~1 ~1.5 ~3GHz BF

~1.5 ~2.5 ~6Gbps CAP

dumb dumb dumb IQ

>100 <100 deployment

Ground segments can easily cost as much as space segment.


Pay for spectrum use

Power consumption in sky

Use a few large dishes, typically dish prices are 5% of annual space segment for > 2 Mbps systems

Focus the beams (don't waste bandwidth transmitting to fishing fleets)

Use effective coding and FEC, raw error rates 1/10000, corrected 1/10M, cf fiber 1 in 10*12.

Fiber/satellite combination

Satellite typical 500ms RTT, versus 30ms for land line.

DESY-MSU 128k fiber $80K/yr

1.5M satellite $120K/y


TAT 90M fiber $12M/yr

Full xpond 140M sat $4M/yr


So there are economic advantages of combining the two technologies. for information on Ka band use in Japan & Europe.

Looking for a killer application.

Cellular style IP access with a large number of spot beams. Use a small number of spot beams to a few sites, but needs a specialized satellite.

$80M for satellite, $80 for launch, $80M for insurance.

Current technology useful in 1999-2000 time frame. In the longer time the cost reduction curve over time will favor fiber cf. satellite.


Video Conferencing - Harvey Newman


Using LBL et al protocols. Integrated packet & codec, connect conference rooms, meeting rooms & desktops: global & local. Workstation packet/codec gateways. Concept of virtual rooms, system of reflectors. Bandwidth efficient protocol suite, good sense of presence 10 fps within 100-200kbps, tunable bandwidth/quality/resource matching. WWW user & network manager interface, includes registry (with profiling), room booking, join/leave conference, net monitoring. Provides sharing of documents at high resolution (e.g. 1200x 1024). Has loopback facility for testing. In production since January 1997. Staff support is to set up and then 80% of the time the user looks after. Next step will focus on using PC systems with Linux. Want to synchronize with industry/commercial standards. NetMeeting does not provide the same functionality - multipoint conferencing (missing for NetMeeting), resolution (missing for White Pine).

Premise is that videoconferncing is an essential par of the tools needed for worldwide collaboration.

Aim is to integrate into daily environment

Distributed Databases - Harvey Newman

Several petabytes of HEP data for CMS in 2005. Replications of database federations, using object oriented software that access database federations. Instittes cluster around ~10 regional centers. Key questions include are is the network bandwidth available, can distributed object orineted databases be made to work in this environment worldwide. Harvey showed that with sufficient bandwidth (1Mbps) local and remote (WAN) commitments are almost identical, but during WAN saturation periods there are factors of 10 difference. Distributed database applications are one of the most network challenging applications.

G7/GIBN - Martin

Global Interconnect Broadband Networks (in particular of G7 countries, but expanding later). Proposal put forward by Gore, approved but with no funding. Tried to convince international operators to provide funding. But funding was in terms of providing capacity but not money. Funding issue came up at January 97 meeting. French delegation took on task of making a proposal on links cost sharing. They produced a European viewpoint document, though there was an attempt to allow it include Japanese-US links. Next meeting is in October. One of topics is to discuss French proposal. Main value is that it is a relatively high level political group so maybe they can make governemnts aware of the importance of interconnecting networks.

TEN-155 - Fernando Liello

TEN-34 has been judged a major success, but it ends in near future. The major weaknesses identified in TEN-34 were: too many suppliers, two core subnets, only best effort IP service (does not exploit full possibilities of ATM service). A proposal has been submitted to the commission. It is called QUANTUM.

Some considerations

ESnet International Issues - Larry Price

Matching access restrictions.

Lot of discussion about what is politically possible. Agreed to raise issue to higher levels also to collect AUPs. Greg Chartrand agreed to gather AUP information. HEPAP might be one place to raise issues since NSF is also represented there. Some of the people who will need to be involved include Jim Leighton, George Strawn and Fernando Liello.

Reciprocal funding

There was contention about the meaning of the statement that Europe is paying much more to connect to the US versus what the US is paying to connect to Europe. It is unclear that the imbalance ($40M vs $2M) is directly related to HEP and thus relevant to ICFA. This imbalance is greater than the imbalance in traffic. It is a highly political issue. One view (enunciated by George Strawn of NSF, earlier in the day) is that it is a consequence of how networking is funded in the US vs Europe (more government support in Europe, and there is less of a central focus in the US).

International access to universities.

Best way to connect would probably to work with Internet 2. Will need to understand AUP. It is possible that vBNS might accept CERN as an International "partner" under the new rules. Larry will discuss with George Strawn whether a Perryman connection between Europe and vBNS would be possible.

UK MCI bottleneck

BaBar people need to identify a programmatic need to get to the UK so that ESnet can get involved.

Foreign primary sites for ESnet

The touchstone site would be for CERN, which could be used as an example and possibly generalized. On the other hand CERN is special in that CERN has DOE funding and so could be special-cased. Could be technical problems, e.g. routing problems if ESnet announce CERN to universities and MCI (e.g. vBNS) also announces CERN to Universities. It was agreed that a technical discussion is initiated to asses such an approach, understand how it might be done, what the costs would be, the implications etc. This technical discussion would need to include at least Denise Heagerty and Jim Leighton (or designate). From the meeting view it was agreed it could be very beneficial to have CERN as a primary ESnet site.

Special routing arrangements

Again there are problems with how things are advertized. For example if JAERI (Japan) wants to get to Garsching it is not possible for ESnet to simply advertize Garsching without advertizing all of the DFN. The Fiber Link Around the Globe (FLAG) initiative which will provide links from Europe to Japan via many countries (going East from UK) with 2*5Gbps could help avoid going via the US.

Next meeting

The normal meeting would be in a year. Should it be sooner? Japan has offered to host the next meeting. It might be considered expensive/difficult (funding) to get there. The intervening video conferences were not an outstanding success due to technical problems. A possibility would be to coordinate something with CHEP98 in Chicago Aug-Sep 98. A draw back might be the emphasis on HEP. In addition we need to consider ICFA-NTF needs. A straw vote for "would you attend if it was held in …" gave: 15 for Japan, Europe 25, Chicago 22. Another straw vote for "how many prefer …" gave: Japan 6, Europe 8, Chicago 8. CHEP might be impractical given the amount of time required by the two meetings, the HEPiX requirements, the organization requirements. The conclusion was that there was a leaning towards Europe. Roman Tirler offered Saclay/Paris. This would help to get Renater people involved, it is also very accessible. It was suggested that May-Jun 98 would be a good time frame for a meeting in Paris.

Transatlantic Requirements - Harvey Newman

US-CERN bandwidth needs:

Requirements Working Group - Harvey Newman

Need to study the impact if the network requirements are not met

Scaling: Storage 5-10, LANs 10-59, CPU 10-50 increase in 5 years.

Requirements (for next 5 years):

ICFA - NTF Discussion





Interim Report Ideas Discussion - led by David Williams

Picture of status (+ recent past & evolution)

Picture of high rate of change (technical, organizational, pressure for privatization …)

Why are we different?

Overview of technology evolution

Remarks on commercial evolution

Service quality across multiple suppliers

Recommendations, choose one:

Global Research Networking - George Strawn US NSF


Early 80's CSnet, mid 80s NSFnet, regionals, connections (56kbps), late 80s T1, early 90s 45Mbps backbone, early 90s international connections, 93 new architecture solicitation (NAPs, RA, vBNS), 95 retirement of NSFnet backbone, establishment of vBNS, 96 high performance connection program (100 universities to connect).

As of 1995, universities (through regionals) had to obtain & fund their own national network backbone network services. The Regionals receive transition support from NSF for 4 yrs declining. Small colleges and K-12 schools libraries & museums can propose to NSF for Internet connections support (typically $15K over 2 years).

Universities awarded vBNS connections agree to ALSO maintain & fund a general purpose Internet connection. Two high performance connection awardees may exchange Internet traffic over vBNS (estimate that 10-15% of a research university's traffic is to another research university).

US Networking

VBNS + high-performance connections (66 awards Spring '97), with another call and anouncements later this fall. Mission agencies networks (DREN, ESnet, NSI/NREN). Internet 2 consortium of universities in US (~110, n.b. there are about 100 research universities with doctoral programs in the US). NGI proposed Presidential Initiative (DoD, DoE, NASA, NIST, NSF) to develop applications and infrastructure for the next generation Internet. US government appropriations is still in process, we do not know whether any additional money will become available.

US Research Internetworking:

NSF Global Research Networking