3/23/98

22 attendees, 8 laptops

LEP scheduled to run to the end of 1999 may get an extra year to end 2000 with extra energy, if the money becomes available. LHC starts mid 2005. With LEP-II hey have enough energy to produce 2 Zs, so luminosity is critical, so the extra time would be very valuable. LHC is going well, have adjudicated contracts for 1/3 of the machine. They are on time and on budget. CERN currently has ~ 2800 people, running down to 2000. Outside users are 6500. The outside users spend 25% of time at CERN. They come from all over the developed world. So they need networking. The budget is not going up, to meet the new challenges. The current CERN budget is about 1000MSF which in current money is less than 1972, and has been stable for many years. Chris liked the NTF report. The recommendation is to do something for the inter-regional connections and let intra-regions look after themselves. Will need to look for allies in other scientific areas, since they will begin to feel the problems soon. Wants us to come up with recommendations for how to address problems, and to provide simple clear explanations of the problems, why we see it first, how others may see it, what is needed to improve the situation with a limited budget.

It took some time to convince ICFA that the effort would be useful, also there was skepticism that anything would come out of the recommendations. There are 5 working groups.

The present status WG is a bit weak on the US situation. The remote regions WG covers the rest of the world outside W. Europe, US, Japan. The report is not available yet. The monitoring WG has tools in place for end-to-end monitoring for between Labs and universities with good characterization of round trip and packet loss. The ICFA needs WG has completed a non-trivial job of characterizing the requirements. The links need to be provided and an executive summary is needed.

Networking is mission critical for ICFA and needs to be seamless world-wide with reasonable performance. The overall performance today is inadequate, We depend on multiple providers. There are trans-oceanic bottlenecks and at IXPs (Internet eXchange Points) and the network is only as good as the weakest link. Physics departments often have difficulties especially with access to Labs.

The first generation applications are email, www, manual data transfer. Second generation is collaboration groupware with audio and video (will go into WWW & email too), automated client access to data servers (file serving such as AFS & DFS), data serving, distributed data bases, "real-time video view of your sub-detector", remote control tools. The third generation is full collaborations. We see examples of 2^nd generation applications, and they have more critical needs for good network performance.

The blame for the poor performance is put on the cost of the trans-oceanic links.

Next will come the final report. Can make 3 recommendations: continue as is, build an ICFA internet, or an intermediate approach with some significant improvements. ICFA chose the intermediate approach and are prepared to lobby to get resources. The ICFA-NTF monitoring group has done a good job, they recommend that it be kept alive. They want to know how much more capacity is needed on the trans-oceanic links.

David had the following personal points. HENP appears to be on the cutting edge in particular in understanding the Internet: its performance & flow and the quality of service NIMI & ANX are interesting examples of approaches.

Expect next 2 years will see enormous Internet growth, Ecommerce is taking, off, the European deregulation taking effect, Worldcom/MCI merger, homes & offices are coming online with cable, xDSL etc. Traffic doubling every 3-6 months.

QoS (Quality of Service) means many things to different people. Intranet gives private links & routers but at a price. VPNs (Virtual Private Networks) give you a sort of virtual network but you need to understand the details. Van Jacobson working on bandwidth broker, there is a proposal to DOE to continue work on this.

Onsite infrastructure will need to be upgraded to work with (take advantage of) the new WAN bandwidths.

There is a view that HEP is arrogant in their requests for bandwidth. Part of the source of this is the contention between the university network management and the demands of the university HEP folks.

We will need to address QoS issues since cannot provision all links (in particular trans-Atlantic) to have sufficient over capacity of links. Once one has QoS then one has to address the issue of cost to request the higher quality. It is the expectation that the fraction of requirements for QoS is only a few percent of the applications. Can we use new, more tolerant applications, e.g. why do we continue to use response time critical applications like telnet.

TEN-34 will be overloaded before 31 June 1998. It use has grown by a factor of 3 since last June. 23TB/month is shipped. Performance has been extremely good. Transit time is 60msec, loss has been minimal. Started losing packets on French (Renater links) since under-provisioned, also at interconnect to UniSource in Germany which is acting as a funnel between commercial and research nets. Otherwise performance is still extremely good. Started at 20% loading, now at 60% expect overload Jun-98. There are 15 suppliers in TEN-34 that makes management difficult.

Competitive tender to provide the successor is in progress. It will cover PoPs, 155Mbps circuits between 4/5/6 countries and circuits to other core countries, also want some form of managed bandwidth service (ATM?) to reserve 10-15% guaranteed bandwidth. It will be an IP service, and US connections to be integrated. This next step beyond goes by name of Quantum (Q for quality of service). A proposal was submitted to the EC (European Community) on 25 Dec 1997. The EC evaluation is complete, DANTE is awaiting committee consideration & formal approval, ITT issued in parallel (should await EC approval, but time scales do not allow luxury, since TEN-34 goes away at end July-98), tenders received 13 Feb 98, tender evaluation in progress. Competition due to liberalization (de-monopolization) is helping price quotes. Bidders include UniSource, Global 1 (Sprint, DT consortium).

Connectivity Internet/DFN/DESY/MSU at 100kbps, then by radio to places in Moscow and by satellite to BINP, Ukraine etc. 80% traffic is for HEP. There is a second link from DFN to Radio-MSU at 100kbps. In future expect to continue to keep DESY connection for supporting HEP traffic for Russia, but not for general traffic. Intelsat recognize can't compete with cable, and advertises satellite to act as a backup for cable systems.

DFN (Deutsche Forschungs Network, the German research Network)/BWIN provides:

• US/CA link via 2*T3
• Multiple TEN-34 lines
• DESY/CERN link went away July-97
• Traffic to US hit limit of 15TB/month.
• T1 fraction of 2*T3 goes to ESnet.
• The BWIN is running PVCs.

Quality over last 6 months for ESnet traffic is satisfactory (US-DE link saturated but "precedence" to give minimal packet loss, ESnet pays what it paid before but now being paid to DFN for a precedence service, rather than paying for a link to DE). It is unacceptable to US non-ESnet sites (50% packet loss). They are happy with European countries and it is excellent within Germany. DESY has hot spots of 6Mbps with average of 2 Mbps (on 48Mbps link). Traffic to MCI (actually from US to Germany) is very bad, the link is saturated for much of day. TEN-34 traffic constantly growing. Losses are between Perryman and Frankfurt.

Expectations and predictions include US-DE DFN working on upgrade that won't be ready before end '98. Precedence feature allows reasonable quality for "mission oriented". Need analysis as well as tools from the ICFA-NTG monitoring WG.

250 sites connected via GARR, 70 universities, 140 Labs & research organizations (INFN, ENEA, INFM etc.), 40 other institutes. It provides high speed IP service, with a virtual link for special applications and a virtual link for testing special applications. GARR-B(roadband) now has 34Mbps (pilot since Oct-96, production 1998) will move to 155 Mbps

ATM cloud with hubs at Milan, Bologna, Rome & Naples. Links to other countries come into hubs. TEN-34 from Milan, US/Perryman via Naples. Want big (45Mbps or 155Mbps) to US. Now have from Milan T1 to Perryman, TEN-34 10-20Mbps Milan-Frankfurt. Hope 45 Mbps via Napoli (ordered in Sep-97). General Internet traffic goes via DANTE. The ESnet T1 is loaded at 80% (averaged over 30 minutes)

Hungary (HBONE) network looks like a star centered on Budapest. Has 10Mbps connection to TEN-34 via ATM. Links are very bad to US (38% packet loss to SLAC), CERN & DESY look good (1%), RAL is 4%.

JANET/UKERNA provides the inland IP service. It is trending to a federal MAN based service. In Scotland there is the"Scottish X-connect". The backbone is migrating to a new provider (C&W) in March '98 (used to be BT). The inland bandwidth is adequate with no major increase planned at this time. The new backbone is called "SJ/III" (SuperJanet/III), based on a core of 155Mbps links (Manchester, Leeds, London, Bristol) with a 155Mbps to the Scottish (MANs at Glasgow/Edinburgh/Stirling and another).

International connection is a Teleglobe link. There was an experimental high speed video conferencing (VC) network which worked well but would not scale, so VC now uses ISDN.

The TEN-34 link is perceived to be good/satisfactory, they are in the dark as to its follow on. Access to CERN or DESY is good (usually), some parts of Europe are hard to reach which does not appear to be a specific HEP problem. There is also a 64kbps leased line RAL-CERN as a fall back for interactive use. It is rarely needed for European use

The US link is provided as an IP service by Teleglobe. It started well but soon became horribly congested and is now essentially unusable (bad packet loss, hundreds of bytes/sec). The major usage is WWW, congestion is USA-UK, second 45Mbps link is coming soon (ordered by UKERNA) but expect to be overloaded in 3 months. The WWW caches proxies have been promoted. People have taken advantage of the 64kbps leased line to CERN for interactive, low bandwidth.

Concerning USA-UK volume related charging, the JISC made decision to recover 2M pounds Sterling. They want to charge for a scarce resource. Shadow charging is being conducted. Institutions to be billed from Autumn '98. Institutions make own arrangements to handle charge. Leading to much discussion and dispute.

They send their vBNS via a Chicago metropolitan network. This is about 30% of the traffic. The other traffic goes via ESnet. Both links are ATM OC3. They do not use any ATM features it is simply IP over ATM. In the US getting IP over SONET costs more.

The plan is to combine the 2 E1 links (today one to NY for MCI Internet, the other to Perryman and only CERN traffic goes via Perryman) to Perryman, will connect via T3 to ESnet, and via 2*T1s to MCI Internet. This is expected to be 3-6 months away. One delay is lack of rack space in the ESnet rack. The rack space is needed among other things is to provide space for the inverse mux.

Reconfiguration at Perryman to replace router with a bigger faster Cisco router (7507) which supports QoS and the hardware/software is capable of up to T3. It has improved traffic management, including weighted fair queuing (WFQ). There is some special handling for virtual room video system. Separate channels with bandwidth bursting by partners. They are getting separate (from ESnet) rack space. There is increased use by IN2P3 (BaBar regional center) so migrating to 1 Mbps. There have been offers for 6-8Mbps with bids in progress (not converged on layout, what lines are merged). The loading has 100% peaks, but less loading when no LEP run. Packet loss rate are 0.1-1% typical on both ESnet and Internet/MCI. There has been an improvement of the Internet/MCI infrastructure. There has also been the start up of the Caltech "primary site" Esnet subnet.

They will have 10**9 events/year, 100+TB data volume/year. First collisions are predicted to be Apr-May '99. All events are kept in an OODB. 500 physicists from over 3 continents, and 70 countries. Lot of reconstruction code is in place. No tools assume a physicist is located at SLAC. Will have regional centers. Packet loss from various sites is very variable from month to month so hard to predict, and almost always the performance is poor.

This requires large efforts locally to by-pass performance problems, e.g. often remote sites have to suck the data over to do their own builds since takes too long via the network to get at the AFS built files. They wanted to do HP builds at a BaBar site (University of Cincinatti) but the network was not up to it, so had to buy machine for LBNL. Terry concurs with the quality characterizations. When does one give up on interactive use. It is probably somewhere above 2.5% for interactive work. It is hopeless above 10%. Harvey says at 3% one sees visible glitches with video conferencing, and at 10% it becomes unusable.

TEN-34 issues include: too many suppliers, two core networks, only best effort IP service ,and the model can't be extended easily. The QUANTUM project will be a single pan-European network, procurement by public tender. Will be a 155Mbps + higher core, maintain the geographic coverage, with an emphasis on QoS, integrate the G7/US/Asia Pacific activity and connection points for foreign NRNs (National Research Networks)

QoS issues include the cost, in particular the ATM overhead and the management costs. VPNs with end-to-end QoS where the implementation can vary between hops. The benefits of QoS are: the generation of an acceptable use policy by helping to segregate/prioritize traffic, with a guaranteed minimum acceptable bandwidth; the building of VPNs which can be offered to smaller communities and provide bandwidth on demand; increased flexibility.

TEN perspectives will give lower cost/Mb but total costs high anyway, the integration of the intercontinental services, a better cost-allocation scheme will allow for greater flexibility. TEN cannot guarantee/obligation to provide QoS to end-users. The VC on demand is far away (couple years in the future).

TEN-34 has proved that sharing an intercontinental line is possible. The TEN design will have in mind the distribution of intercontinental services. There are still funding questions. The intercontinental PoPs will simplify management

BINP (Budker Institute of Nuclear Physics - Novosibirsk) has a 128kbps landline to KEK which was started March 1^st 1998. It will be connected to ESnet. Currently to BINP to Japan the are seeing 130ms round-trip times, and 0% packet loss (c.f. via DESY satellite was 1000ms and 40% loss). IHEP (Beijing) 64kbps landline to KEK will upgrade to 128kbps. KEK to ESnet is 1.5Mbps, NACSIS to Chicago NAP 45 Mbps (prior to Oct-97 was 6Mbps and after the upgrade it saturated in 2 weeks) which will upgrade to 155Mbps. NACSIS thru US to Europe (DANTE/TEN-34) at 2Mbps.

HEPNET-J (AS2505) is built on a 155Mbps backbone PVC with sites having their own HEP class C subnets. Will upgrade to 270Mbps in CY98. It forms a VPN.

QoS is obtainable only by guaranteed bandwidth for HENP using the HENP dedicated PVC via CIR (Committed Information rate). Frame relay could provide this but only up to a few Mbps.

In the HEP dedicated lines the costs are shared equally. KEK wants to maintain some dedicated bandwidth to CERN & DESY. They are thinking of converting KEK-ESnet 1.5Mbps leased line into a PVC in 1.5Mbps (CIR = 1Mbps) using Frame-Relay (or ATM) and to have another PVC for connecting CERN & DESY.

$200-250M to launch. $100M for launch vehicle, $100M for satellite the rest for insurance. 100 transponders of 36Mbits each. It may be possible to go to 250 transponders at 60Mbps each (limited by geometry, solar power and spectrum availability). Used for telephony in 1970's but mainly lost market with undersea cables which have better bit error rates and lower delays. Now concentrate on broadcasting. Internet may be interested for multi-cast. Low earth orbit satellites avoid delays. Globalstar (Loral, due in 2000/2001), Iridium (Motorola, due later this year) are market leaders are they are aimed at the cellular phone market (which is volume charged). There is no killer application for high bandwidth. Moving from Ku (10-20), Ka (20-30) to V band (40-60 GHz). Also lose a few hours per year when sun is directly behind the satellite. Does not look super urgent for ICFA to pursue this approach. We will follow products as come to market and look for suitable applications. May be relevant for stand-alone connectivity (1-1.5Mbps) for remote regions, but volume charging may be a problem.

David wanted to explore the feasibility of "broadcasting" the seminars which take place in our major Labs. Basic idea is that most big Labs have arrangements for recording and even transmitting audio & video of major seminars. The technology to broadcast is clearly all available, the only issue is cost. They talked to Digital Europe and now are talking to Williams Communications. You can get a seminar to an uplink. Uplink can be leased or bought (plus a license to operate).

Use satellite where phone cost $5/min, and person gets $200/month. Three satellites cover globe. Get one hop to any site with 64kbps @ $20K/annum. $2M at $280K/annum plus $100K for dish. But if try to provide all Internet service this way then will probably saturate.

What should ICFA support? It could be useful for web caching with broadcasting (advertised by IntelSat). For remote regions could have a hub station. The hard decision would be which remote region to support for a pilot. Probably cannot get help from other agencies since it is mainly for HEP.

Monitoring related topics:

• What is the size of ICFA networking & how to track, deploy MRTG more widely, something else?
• How much sub-ocean bandwidth needed to make the situation more reasonable?
• Is any specific action needed to keep ICFA-NTF monitoring group operational in the medium term?
• What should the monitoring group do next?

Interactive traffic

• Can we identify traffic that needs priority.
• What is needed in order that international interactive traffic can be handled sensibly.

Fat Pipes

• Can we work out a sensible proposal with other disciplines to deploy ATM paths on say 34Mbps & 45Mbps. End point needs paths to major sites in Europe.

Propose some applications that make sense for satellite

Recommendations

It was agreed that one of the most important activities to pursue is to maintain the coordination built up between the HENP Labs to identify networking bottlenecks., in particular so resources can be effectively deployed. To enable this we should actively continue the monitoring activities, especially since the network situation is extremely fluid. In particular ICFA support of the SLAC and HEPNRC efforts was encouraged including ongoing and extended support from DOE/MICS. It was also recommended that similar support should be provided for monitoring activities in Europe and Japan.

Activities recommended to be pursued by the monitoring group included:

• Further analysis & understanding of the data by using grouping, probably most of this will come from SLAC & HEPNRC (one group mentioned was the TEN-34 partners);
• Getting a characterization (e.g. volumes by application, source/destination pairs etc.) of the traffic by looking at traffic types at a few major Lab borders and at least one university. Olivier Martin & Charles Granieri are working on this at CERN, FNAL, SLAC and Caltech hoped to also look at this.
• DESY, FNAL, CNAF, KEK and RAL agreed to provide public access to MRTG reports of traffic volumes at their border routers. SLAC, CERN and RMKI already provide this. The purpose is to get an idea of how much traffic goes to & from the major HENP labs. RAL is going to try get a handle on how much of its traffic is HEP.
• Several Labs expressed interest in providing traceroute maps to facilitate an understanding of the Internet topology.

Desirability of Solutions, balancing usefulness and political correctness

Solutions (0)

• Expect to spend money/FTE on achieving & maintaining an appropriate level of national, international & intercontinental network performance

Solutions (1)

• Prioritized traffic on general-purpose links
• Germany - US, Japan - US? UK - US? Italy - US? France - US??? TEN-155 - US?

Solutions (2)

• HEP - manage links (with prioritized interactive traffic)
• CERN - US, CCIN2P3 - CERN - US, KEK - IHEP, KEK - BINP, KEK - US, DESY - MSU

Solutions (3)

• Satellites, consider where competitive (or only) option (e.g. remote regions)

Solutions (4)

• Investing (politically/financially) in the general-purpose internet.
• Academic & research (TEN-34 etc.)
• General purpose commercial

The ICFA meeting is from 23-29 July in Vancouver. W must deliver the final report by Wednesday July 1 and ideally one week earlier. Must have a good draft by May 18^th and ideally by previous Friday (March 15^th). Final reports from working groups (WG) by Friday May 1^st (5 weeks time).

The Present Status group is pretty much done apart from a US update. Dave Martin will try and provide information on NGI/Internet II etc. The Remote Regions WG should start work, but it is unlikely to have anything written by the deadline. The Monitoring WG report needs to include some of the more recent information including that presented at this meeting. The Requirements WG needs a more complete rport with references and possibly a formula.

The recommendations are:

The study has been very valuable, the situation is extremely fluid, we must continue coordination between the Labs and the nets on which we depend.

The monitoring needs to continue, it provides much needed facts to understand the situation and set expectations, to argue our case, to keep our house in order,to identify problems and bottlenecks and indicate how to resolve, it provides a base for a medium (1-3 year) strategy, allows us to optimize existing networks and improve performance. The major labs, HEPNRC and the agencies should at least maintain the present level of resources.

More work is encouraged to provide a better understanding of remote region networking issues. It needs more active involvement of the people from these regions.

The Present Status WG work is seen as very valuable and should be kept up to date.

We need to build and keep refining a model for exisiting needs and future growth. We need to distinguish data transfer requirements (AFS, DFS, FTP, especially HEP to HEP) from more collaborative use.

Specific ideas for improvement include: need to address the bottlenecks on many trans-Atlantic routes, but no single route helps multiple communities, so no single investment in a "Fat Pipe" looks like an obvious winner, the most desirable opportunity is taking some small priority bandwidth on an Academic and Research (A&R) connection.

Continue coordination activities, possibly in the same forum

Continue activities of the NTF Monitoring group

Better define & quantify the model for future growth, including the requirements for essential functions vs. physics data transfer/distribution and their relative priority

Understand how our activities relate to other disciplines, how they are similar and different.

Encourage a better understanding of the network status & possible development in the remote regions.

Keep our house in order by providing ongoing attention in the midst of a fluid situation.

Engage in feasibility projects to improve the current situation and future growth.

Recommendations on transoceanic issues:

• How best to phrase the recommendations for improving the trans-oceanic bandwidth, what is worth new money? They (ICFA) want a timetable & 12-month program of work, so what is feasible for a June final report?
• Encourage cooperation with other disciplines & agencies in sharing the transoceanic bandwidth (in a way useful for HEP).
• Consider urging the funding agencies to provide priority HEP service on transoceanic links either using reserved bandwidth (QoS) or separate links.

Encourage all organizations, including labs, A&R nets, funding & regulatory agencies to work to extent possible to promote end-to-end connectivity.