ESnet International & ICFA-NTF Meeting
Santa Fe, September 8-10, 1997
Rough Notes Taken By Les Cottrell
International Status
ESnet Update
Report from ICFA
Questions Raised by Working Groups
International Nuclear Safety Program
Introducing Submarine Cables
Satellite Update
Video Conferencing
Distributed Databases
GIBN
TEN-155
ESnet International Issues
Transatlantic Requirements
Requirements Working Group
ICFA-NTF Discussion
Interim Report Discussion
Global Networking
TEN-34 - Fernando Liello
TEN-34 see http://www.dante.net/ten-34, for DANTE see http://www.dante.net/
Mixture of ATM & leased circuits
Combination of managed services & IP links
ATM policy is only academic & research
IP includes commercial on best effort basis
Work item for international connectivity, but no funding
Economics of sharing are improved by TEN-34
Currently 68Mbps NORDUnet
16Mbps SURFnet
53Mbps Janet
TEN-34 Progress
2-8Mbps CERN/Switch
50% costs for same capacity
Loading Paris London (only 6Mbps ATM) also Netherlands-Germany (22 Mbps) 40-50% loaded
London-Stockholm 20-30% loaded
In general ATM lightly loaded, IP is heavily loaded in some cases, partially due to commercial traffic
Challenges
TEN-34 commitment only until Aug-98
Too many suppliers, too expensive
Liberalization in Europe Telecomm Jan 1 1998.
Issues
US access restrictions do not match European policy
Different US discipline orientations
Limited funding
STAR-TAP
Co-Funding
Europe-US 305 Mbps
Europe - US lines $40M / yr
NSF contribution $2M?
Funding question remains
Hard to justify
NSF Solicitation
DANTE part of University of Illinois submission
Will utilize existing lines & TEN-34
Conclusion
GAP in funding remains
ESnet contribution ($100K/yr for Italian co-funding)?
US political will to solve the problem?
DFN Status - Martin Willhelm
WIN (broadband network) in operation for more than a year. AUP does not allow commercial transit. Has a running ATM backbone used for transmission of IP traffic. Have 10 ATM nodes with 350 institutions from research & education. Have got more sites connected than expected. 10 sites at 155Mbps, 64 at 34Mbps. Overall performance and stability is quite good. Contracted to Deutsche Telecomm. Still have a small part of X.25 network to be phased out next year (it costs considerable money, and some users are hard to move). Will be offering native ATM connections. Have some pilots running. Hope to offer at end of next year to some universities. Have requests to put voice on. Preparations for next phases have started. In next 2-3 years hope for transition to OC12 network. Have pilot running today. Have TEN-3 connection at 34Mbps which is quite satisfying. Currently carrying 34Mbytes/month. They have a guarantee that up to 400Mbps peaks they will have good performance. They now carry 300Mbps peaks, so may expect problems in 6 months.
US connection at Perryman (MCI clocation space) , MD with 90 Mbps (2*T3 on different cables, one on PTAT-12, one on PTAT-13, cost is about 22M DM/yr, i.e. about 1/3 cost & bandwidth from Europe to US). Deutsche Telecomm manages 2*Cisco 7513 routers at Perryman. They have redundancy (2 routers, 2 switches) also can dial in out of band, as well as a contract with MCI to provide technician support. Have transit agreement with MCI to US and rest of world. Have a T1 connection to Esnet. Will be upgrading to T3. Have rented additional space for other parties such as DANTE so they can colocate there also. Not considering a presence at STAR-TAP, no need, can't afford. NGI considering 3 interconnect points (W Coast, Chicago, E Coast), so DFN could connect to E Coast.
Italy - Antonia Ghiselli
GARR-B follow on (Broadband)
Requirements for US connections
Need 4Mbps or 155Mbps with peering to ESnet, NASA, Internet 2, IP general Internet, G7/GIBN, others.
Looked at Telecom Italia ordered 45Mbps on Columbus 2 ordered July 97, delivery ?)
Via TEN-34: 10-20Mbps thru Milan-Franfurt (0ct-97)
Bunch of 2Mbps
Proposal for US link:
Japan - Yukio Karita
http://www.kek.jp/~karita/santafe-report.html
HEPnet-J
KEK & most universities on HEPnet-J, using PVC
International lines
HEP dedicated KEK-ESnet 1.5Mbps last Saturday very good
KEK-IHEP upgrade from 64kbps to 128kbps
KRK-BINP 128Kbps also slowly progressing
NACSIS general A&R lines
NACSID - US SprintLink 6Mbps (heavily saturated, very bad, unusable) -> 45Mbps line
NACSIS Europe (DANTE) 2Mbps becoming bad
Two major programs for US & TEN-34
US universities
Lot of complaints from researchers in US universities
Have been expecting routes to US universities to be given thru Esnet. They wll stop using
NACSIS-US line until upgraded to 45Mbps so get good connectivity via Esnet. TEN-34 has not been good for Japan. Had assumed termination of NACSIS-US line would be moved from Europanet to TEN-34, but was wrong. Europanet link goes thru saturated link Geneva - London. Would like to move termination of NACSIS line to TEN-34, but Unisource insists on charging a fee for the access of $34K/month. Note that NACSIS is already paying for the full circuit if the NACSIS/DANTE line (Europeans pay nothing), also note DANTE does not charge for connections. Thus NACSIS is relocating termination point from Amsterdam to London to avoid paying $34K/month. But relocation takes time. Afraid London-Geneva gateway will be saturated before the relocation is done.
UK - Paul Jeffreys
UKERNA (an independent government funded non profit making company) operates a 155mbps national academic network.
UK Europe
TEN-34 service started on 18th March
Added parts of TEN-35. RTT halved, packet loss much better
Performance to CERN is bounded to deteriortae
Conutries are committed to cancelling their rented lines
UK-US
1995 4Mbps for entire UK
Early 1996 8+6 * 1.54Mbps)
Mar 1997 t3 (45 Mbps) WWW traffic tken onto old connection
Mid july - early Aug 1997 fiber dies had to use satellite.
Harvey says up to 85% utilization packet loss is small, then starts to rise
Canada - Dean Karlen
CA*net has a linear 10 Mbps link with 4 links to US (aggregate of 90Mbps). One of which is a direct link to Esnet to bypass MCI problems (e.g. at West Orange).
CA*net II will use OC3 with GigaPOPs at Province capitals. Has connection to STAR-TAP. Does not replace existing commercial networks, organizations must maintain a separate link to the Internet. <10% of University traffic will be diverted to CA*net II
AUP Tier A: Universities & Labs. allow transit traffic. See STAR-TAP as an international exchange point.
Very interested in full peering at STAR-TAP, expect all major research nets will meet at STAR-TAP.
ITER -
They are getting good connectivity via ESnet to Germany & Japan. Would like to improve connectivity to Russia, in particular the Khurchatov institute in Moscow. ITER ends July-1998. Have not selected a site yet. Hopefully then can move into a construction stage. May enter a 3 year pre-construction phase. Target is by 2009 many countries can connect to the ITER database and use interactive control.
DESY - Michael Ernst
DESY has 34Mbps to BWIN. Still has connections to CERN link, cancelled mid '97 and connect via TEN-34, and terminate wide-area DECnet. US/CA connectivity via 2 * 43Mbps links.
Quality:
International is satisfactory to US/CA, excellent over TEN-34
US/ESnet 2-10%, US/CA 10-20%, Japan 5-10%, CERN/CH 0-5%, UK (RAL) 0-5%, FR (CEA) 0-5%), Germany 0-1%.
Vol.ume DESY-International 180GB, Int.-DESY 150GB, national traffic is similar. Leighton (using pathchar) sees losses are occurring over transatlantic link (from US to Germany). Michael showed graph (created using the MSRP tool that looks at router statistics - several others showed graphs using this tool which looks extremely useful for measuring utilization) showing that transatlantic link is saturating for several hours each day.
In Hamburg have an FDDI MAE network, will be changed over to a central ATM switch network.
ESnet - Jim Leighton
Japan
International Issues
International connectivity on East Coast
Upgraded ESnet DFN, bandwidth, cost & bandwidth guarantee TBD.
CA*net T1 to PPPL may be obsoleted by ESnet connection to Chicago NAP, interest in access to CERN.
CERN carry as ESnet primary site?
Connection to Caltech underway (executive at Caltech will write letter).
DANTE plans to establish a POP in Perryman (good news).
General university access (needs written letter from high up in University stating that they are cognizant of the special connection and nobody on the campus has a problem with it).
STAR-TAP access is coming.
VBNS
There are only 16 Universities with vBNS access today. Jim was unable to get a schedule of when the other 30 odd sites will be added.
VBNS moving ahead with Internet 2 and connections program.
Bad list of Universities from ESSC study was (+ already there, - soon (we hope)):
The poor list was:
See http://nicewww.cern.ch/~davidw/public/ICFAESSC/outlinee.htm
David showed a lot of SLAC generated 180 day plots of performance and narrated the observations. He said they were extremely useful to understand what is going on. These were presented at the Hamburg ICFA meeting.
Questions raised by Working Groups
Monitoring WG
Requirements WG
Status WG
Remote WG
International Nuclear Safety Programs - Greg Chartrand
New program:
Big challenges are coordination, contracting, customs, communications, travel, building trust, demonstratin relevance, fulfilling commitments, achieving safety.
Greg is looking for collaboration and exchange of information.
Introducing Submarine cables - David Williams
See: http://nicewww.cern.ch/~davidw/public/ICFASubCables/ppframe.htm
A major limitation (on number of pairs and installation cost) is for the repeaters. Underlying costs are halving every 18 months. Expect factor 100 in next 10 years. There are beginning to be non-consortium run cables appearing, maybe this will bring down costs due to competition. There appears to be a big markup (10 to 20 times) of cost of installation against what cable owners sell capacity.
Satellite Update - Hans Frese
Two main uses of satellite capacity:
Orbits
Frequencies
C Ku Ka Band
4/6 12/14 20/30GHz CF (only C band is rain tolerant)
~1 ~1.5 ~3GHz BF
~1.5 ~2.5 ~6Gbps CAP
dumb dumb dumb IQ
>100 <100 deployment
Ground segments can easily cost as much as space segment.
Pricing
Pay for spectrum use
Power consumption in sky
Use a few large dishes, typically dish prices are 5% of annual space segment for > 2 Mbps systems
Focus the beams (don't waste bandwidth transmitting to fishing fleets)
Use effective coding and FEC, raw error rates 1/10000, corrected 1/10M, cf fiber 1 in 10*12.
Fiber/satellite combination
Satellite typical 500ms RTT, versus 30ms for land line.
DESY-MSU 128k fiber $80K/yr
1.5M satellite $120K/y
TAT 90M fiber $12M/yr
Full xpond 140M sat $4M/yr
So there are economic advantages of combining the two technologies.
http://www.spotbeam.com/ for information on Ka band use in Japan & Europe.
Looking for a killer application.
Cellular style IP access with a large number of spot beams. Use a small number of spot beams to a few sites, but needs a specialized satellite.
$80M for satellite, $80 for launch, $80M for insurance.
Current technology useful in 1999-2000 time frame. In the longer time the cost reduction curve over time will favor fiber cf. satellite.
See http://kronos.lerc.nasa.gov/acts/acts.html
Video Conferencing - Harvey Newman
See http://Sgil3conf2.cern.ch:8181/
Using LBL et al protocols. Integrated packet & codec, connect conference rooms, meeting rooms & desktops: global & local. Workstation packet/codec gateways. Concept of virtual rooms, system of reflectors. Bandwidth efficient protocol suite, good sense of presence 10 fps within 100-200kbps, tunable bandwidth/quality/resource matching. WWW user & network manager interface, includes registry (with profiling), room booking, join/leave conference, net monitoring. Provides sharing of documents at high resolution (e.g. 1200x 1024). Has loopback facility for testing. In production since January 1997. Staff support is to set up and then 80% of the time the user looks after. Next step will focus on using PC systems with Linux. Want to synchronize with industry/commercial standards. NetMeeting does not provide the same functionality - multipoint conferencing (missing for NetMeeting), resolution (missing for White Pine).
Premise is that videoconferncing is an essential par of the tools needed for worldwide collaboration.
Aim is to integrate into daily environment
Distributed Databases - Harvey Newman
Several petabytes of HEP data for CMS in 2005. Replications of database federations, using object oriented software that access database federations. Instittes cluster around ~10 regional centers. Key questions include are is the network bandwidth available, can distributed object orineted databases be made to work in this environment worldwide. Harvey showed that with sufficient bandwidth (1Mbps) local and remote (WAN) commitments are almost identical, but during WAN saturation periods there are factors of 10 difference. Distributed database applications are one of the most network challenging applications.
G7/GIBN - Martin
Global Interconnect Broadband Networks (in particular of G7 countries, but expanding later). Proposal put forward by Gore, approved but with no funding. Tried to convince international operators to provide funding. But funding was in terms of providing capacity but not money. Funding issue came up at January 97 meeting. French delegation took on task of making a proposal on links cost sharing. They produced a European viewpoint document, though there was an attempt to allow it include Japanese-US links. Next meeting is in October. One of topics is to discuss French proposal. Main value is that it is a relatively high level political group so maybe they can make governemnts aware of the importance of interconnecting networks.
TEN-155 - Fernando Liello
TEN-34 has been judged a major success, but it ends in near future. The major weaknesses identified in TEN-34 were: too many suppliers, two core subnets, only best effort IP service (does not exploit full possibilities of ATM service). A proposal has been submitted to the commission. It is called QUANTUM.
Some considerations
ESnet International Issues - Larry Price
Matching access restrictions.
Lot of discussion about what is politically possible. Agreed to raise issue to higher levels also to collect AUPs. Greg Chartrand agreed to gather AUP information. HEPAP might be one place to raise issues since NSF is also represented there. Some of the people who will need to be involved include Jim Leighton, George Strawn and Fernando Liello.
Reciprocal funding
There was contention about the meaning of the statement that Europe is paying much more to connect to the US versus what the US is paying to connect to Europe. It is unclear that the imbalance ($40M vs $2M) is directly related to HEP and thus relevant to ICFA. This imbalance is greater than the imbalance in traffic. It is a highly political issue. One view (enunciated by George Strawn of NSF, earlier in the day) is that it is a consequence of how networking is funded in the US vs Europe (more government support in Europe, and there is less of a central focus in the US).
International access to universities.
Best way to connect would probably to work with Internet 2. Will need to understand AUP. It is possible that vBNS might accept CERN as an International "partner" under the new rules. Larry will discuss with George Strawn whether a Perryman connection between Europe and vBNS would be possible.
UK MCI bottleneck
BaBar people need to identify a programmatic need to get to the UK so that ESnet can get involved.
Foreign primary sites for ESnet
The touchstone site would be for CERN, which could be used as an example and possibly generalized. On the other hand CERN is special in that CERN has DOE funding and so could be special-cased. Could be technical problems, e.g. routing problems if ESnet announce CERN to universities and MCI (e.g. vBNS) also announces CERN to Universities. It was agreed that a technical discussion is initiated to asses such an approach, understand how it might be done, what the costs would be, the implications etc. This technical discussion would need to include at least Denise Heagerty and Jim Leighton (or designate). From the meeting view it was agreed it could be very beneficial to have CERN as a primary ESnet site.
Special routing arrangements
Again there are problems with how things are advertized. For example if JAERI (Japan) wants to get to Garsching it is not possible for ESnet to simply advertize Garsching without advertizing all of the DFN. The Fiber Link Around the Globe (FLAG) initiative which will provide links from Europe to Japan via many countries (going East from UK) with 2*5Gbps could help avoid going via the US.
Next meeting
The normal meeting would be in a year. Should it be sooner? Japan has offered to host the next meeting. It might be considered expensive/difficult (funding) to get there. The intervening video conferences were not an outstanding success due to technical problems. A possibility would be to coordinate something with CHEP98 in Chicago Aug-Sep 98. A draw back might be the emphasis on HEP. In addition we need to consider ICFA-NTF needs. A straw vote for "would you attend if it was held in " gave: 15 for Japan, Europe 25, Chicago 22. Another straw vote for "how many prefer " gave: Japan 6, Europe 8, Chicago 8. CHEP might be impractical given the amount of time required by the two meetings, the HEPiX requirements, the organization requirements. The conclusion was that there was a leaning towards Europe. Roman Tirler offered Saclay/Paris. This would help to get Renater people involved, it is also very accessible. It was suggested that May-Jun 98 would be a good time frame for a meeting in Paris.
Transatlantic Requirements - Harvey Newman
US-CERN bandwidth needs:
Requirements Working Group - Harvey Newman
Need to study the impact if the network requirements are not met
Scaling: Storage 5-10, LANs 10-59, CPU 10-50 increase in 5 years.
Requirements (for next 5 years):
ICFA - NTF Discussion
Monitoring
Status
Remote
Requirements
Interim Report Ideas Discussion - led by David Williams
Picture of status (+ recent past & evolution)
Picture of high rate of change (technical, organizational, pressure for privatization )
Why are we different?
Overview of technology evolution
Remarks on commercial evolution
Service quality across multiple suppliers
Recommendations, choose one:
Global Research Networking - George Strawn US NSF
http://sgi6.hep.anl.gov:8001/strawn.ppt
History
Early 80's CSnet, mid 80s NSFnet, regionals, connections (56kbps), late 80s T1, early 90s 45Mbps backbone, early 90s international connections, 93 new architecture solicitation (NAPs, RA, vBNS), 95 retirement of NSFnet backbone, establishment of vBNS, 96 high performance connection program (100 universities to connect).
As of 1995, universities (through regionals) had to obtain & fund their own national network backbone network services. The Regionals receive transition support from NSF for 4 yrs declining. Small colleges and K-12 schools libraries & museums can propose to NSF for Internet connections support (typically $15K over 2 years).
Universities awarded vBNS connections agree to ALSO maintain & fund a general purpose Internet connection. Two high performance connection awardees may exchange Internet traffic over vBNS (estimate that 10-15% of a research university's traffic is to another research university).
US Networking
VBNS + high-performance connections (66 awards Spring '97), with another call and anouncements later this fall. Mission agencies networks (DREN, ESnet, NSI/NREN). Internet 2 consortium of universities in US (~110, n.b. there are about 100 research universities with doctoral programs in the US). NGI proposed Presidential Initiative (DoD, DoE, NASA, NIST, NSF) to develop applications and infrastructure for the next generation Internet. US government appropriations is still in process, we do not know whether any additional money will become available.
US Research Internetworking:
NSF Global Research Networking
Issues
Conclusions